Getting to Impact Through Evaluation

by Matrix Admin | Apr 04, 2011
Evaluation can be a daunting topic for foundations of all sizes—at least the way most people think of evaluation. So why encourage small foundations with already limited time and money to take on evaluation?

That’s easy: Evaluation is a key element for foundation managers who want to increase the impact of their giving. Getting to impact through the work of a foundation requires good planning, intentional work toward a goal, and a drive to keep learning and improving—and evaluation is a critical tool along the way.

Although evaluation may have been overwhelming or unproductive for your foundation in the past, it doesn’t have to be going forward. It can be simple and straightforward.

Even on the smallest scale, evaluation can:
  • Provide information for decision making
  • Measure progress and motivate your board and staff
  • Help you to be as effective as possible with limited resources
  • Help you to be transparent and credible
  • Help you to learn, plan, and improve all aspects of your work

Set the Stage

If you sense that your key players doubt the value of evaluation, tackle the challenge head on: Ask hard questions about the foundation’s impact, introduce the naysayers to others who put evaluation to good use, and share materials to educate them about simple, straightforward evaluation methods.

For support, see Exponent Philanthropy’s primer Getting to Impact: Why Evaluation Is Key. With buy-in in place, you will find evaluation to be significantly easier.

In addition to buy-in, every evaluation needs a manager to keep things moving forward. Your foundation will benefit by selecting one individual or a team of individuals as overseers—typically those who are curious, strategic, and focused on improvement. Some foundations even hire a consultant to fill this role.

“Evaluation is an essential process at Hau’oli Mau Loa Foundation,” says Exponent Philanthropy member Janis Reischmann of the Hawaii-based foundation. “It does take effort, and we’ve made lots of missteps. But we keep at it because we believe that learning from what we’re doing, together with the organizations we fund, is an essential element of being an effective funder.”

Create Your Road Map

Before beginning any evaluation, your foundation must be clear about the impact it hopes to achieve and how it plans to get there. Only then will you know where to target an evaluation.

“It sounds obvious,” says Exponent Philanthropy member Gerald McCarthy of Virginia Environmental Endowment, “but it is important to have a clear idea of what your foundation wants to accomplish. Then, if you add in evaluation and thoughtfully put the findings to work, results almost always follow.”

For help to articulate your intended impact and how you plan to achieve it, see Exponent Philanthropy’s primer Getting to Impact Through Planning.

Target Your Evaluation

To introduce evaluation as a tool, start by selecting a target that makes sense for you right now. Although there are universal reasons to always make evaluation part of your work, there are often special reasons to explore particular aspects of your work. The goal is to target those aspects that are currently most important so you can use your limited staff, time, and assets wisely.

To choose your evaluation target, follow three steps:

Determine your motivation. Foundations engage in evaluation for many reasons. In fact, to some degree, your foundation may be motivated by all the reasons that follow:
  • Guide future strategies for new or pilot programs
  • Improve ongoing foundation strategies
  • Build grantees’ evaluation capacity
  • Make good funding decisions
  • Hold grantees accountable

As you might imagine, evaluation will look very different if your goal is grantee accountability versus foundation impact. Although you may initially think about evaluation as a way to monitor how funds were spent or decide the next grant, we encourage you to engage in evaluation for a deeper purpose: to improve your own practices.

Stay within your sphere of influence. Perhaps the most common evaluation pitfall is a desire to make claims that are beyond our control as grantmakers. But many factors are at play in large-scale social change, from charitable giving to a nation’s economic health.

You’ll better understand your foundation’s role in creating change if you target your evaluation on that which is within your sphere of influence.

Set priorities. Prioritize what is most important for the foundation to know now. “We started out small, evaluating one thing and building on that,” says Exponent Philanthropy member Colleen O’Keefe of Sauer Children’s Renew Foundation in Minnesota. “This made the process manageable.”

A particular topic might be important to the foundation if it is:
  • A new funding area for the foundation
  • A significant portion of the foundation’s spending
  • A program area that seems to be floundering
  • Something about which the board cares deeply or something that piques its interest
  • Something that could greatly improve the foundation’s ability to operate more effectively

Be sure you’re open to change in a particular area. For example, before evaluating the usefulness of your foundation’s website, consider whether you have the desire and capacity to update the site in response to your findings.

Ask the Right Questions

Once you’ve targeted your evaluation on a particular area of your work, you can move to the next step: formulating a few evaluation questions.

Strive to ask questions that are specific; they’ll point you to the information you need to answer them. For example, ask this: Because of our grants for health education, do more teen mothers receive prenatal care? Not this: Do our grants for health education make a difference?

Foundations that are focused in their giving are typically able to ask more specific questions.

Track Only What You Need

Many funders get stuck here: the point at which you must attempt to answer your evaluation questions. As Warren Buffett accurately notes, “It is easy to tell if you are succeeding in business: You make money. In philanthropy, measuring performance can be fiendishly tricky and take a lot longer.” (The Economist, June 29, 2006)

To track what you need to answer your evaluation questions, follow three steps:

Choose indicators. Also known as metrics or measures, indicators are the pieces of information that help you answer evaluation questions. Indicators can be quantitative (usually packaged as a number or percentage) or qualitative (providing a context for quantitative data).

Choosing helpful indicators will take some practice; there is no set of right indicators. The good news is that you will learn something from whatever you choose to track—even if you only learn that you want to track something different next time! Plus you’ll develop the good habit of assessing your work based on real information and not mere assumptions about what is working and why.

What information do you need to answer your evaluation questions? Prioritize the indicators that can be measured or assessed with relative ease, using internal sources or the help of grantees.

Build consensus with grantees. Although the evaluation is yours, grantees are critical allies in providing the data you need—and engaging grantees at this point in the process can go far. Says Exponent Philanthropy member Jean Buckley of Tracy Family Foundation in Illinois, “We find that when we bring the grantee into the process of selecting indicators, the grantee feels valued as a partner and consequently takes increased ownership of the indicators and the results.”

Before you meet with grantees to discuss your evaluation plans, remember two things: First, your grantees’ main purpose is to run programs—not do evaluation work. Evaluation requirements often differ from funder to funder, and grantees face little or no funding to cover the costs of evaluation. Second, grantees can fear that unfavorable results will prevent future funding. Help them understand that your evaluation is about learning with them and from them, not about judging.

Set up a data collection work plan. Once you have chosen your indicators, it is time to plan how to collect the appropriate information. Use a simple work plan to identify what you’ll collect, from which sources, how, when, and who will be responsible.

There are several ways to collect data, but foundations most often use grant reports; surveys (online or written); site visits; case studies; or interviews (individual or group). Beyond formal tracking, some foundations also find it useful to meet with grantees to gather less formal data. Be sure to place your data into a spreadsheet or other tracking tool so you can examine the results with ease.

Learn the Important Lessons

You might think that this step is obvious. Unfortunately, even large foundations with many staff gather data yet never glance at the information. Although learning might not be easy, it is a must if you want to increase your impact. How can you ensure that your foundation approaches its evaluation ready to learn from the results? Follow these steps:

Have the right attitude. See your work as an experiment and approach it with curiosity. It takes the pressure off in a healthy way and can boost your morale—and your impact—over time. And be gentle with yourselves and your grantees. No one has all the answers, especially considering the complexities of most social challenges. It’s unfair to expect that your foundation will get it right the first time, and the same is true for your grantees.

“Sometimes I feel myself cringe when I see negative results,” says O’Keefe of Sauer Children’s Renew Foundation, “but I have to remind myself that even negative results are useful for helping us adjust our course.”

Carve out time for reflection. Foundation board members are more likely to care about evaluation if they are encouraged to think about the results. If not moving directly to a conversation about using what you have learned (see the next section), set a nearby date on which you will have this very important discussion, even if it means scheduling a call between board meetings.

Use Your Learning for Greater Impact

As is the case with most lofty questions, it is important to recognize that it’s easier to ask a question than to act on it, because acting often requires you to make changes. It takes effort and energy and time to change—all of which small foundations typically find scarce.

Use what you have learned by following three simple steps:

Use your learning within your foundation. Consider the following questions:

  • How does our learning validate our plan for impact or lead us to change it?
  • How does our learning validate our evaluation work or lead us to change it?
  • Could organizational changes (e.g., board changes, staff changes, budget) lead to greater results?
  • Are we making a substantial difference in our focus area, or does our learning suggest we change it?
  • If there are many changes to make, which have the potential to increase the foundation’s impact most?

Incorporate your decisions into your next round of planning and implementation. Some changes will be small and can be made immediately, but others will be best addressed at the next board retreat or during the next round of grantmaking. Over the years, as your foundation repeats the cycle of evaluating—learning—using, it will become second nature to base your decisions on data.

Exponent Philanthropy member Cathie Gura of The Children’s Guild Foundation in New York underscores the strategic shifts that her board has made since using data to guide its decisions: The foundation is more proactive, builds skills among nonprofits, and now funds on a rolling basis to engage matching state and federal funds.

Use your learning with your grantees. Small foundations that enjoy trusting relationships with grantees are in a prime position to extend learning beyond the foundation. For starters, consider one-on-one discussions with grantees about evaluation results. Not only is the foundation likely to gain valuable insights from grantees, the discussion will easily shift to what the grantee is learning about its work. This is especially true if the conversation is separate from a discussion on future funding.

In addition, gather grantees to discuss trends and/or troubleshoot problems. Sometimes a foundation’s evaluation reveals important trends or challenges in a field that could be best addressed by group problem solving.

Use your learning with other funders and the broader field. Another way to use your learning is to share it with others in the same field. Whether you develop formal groups to share the information or share it informally over coffee, the important detail is that the information gets out. Not only can others benefit from what you are learning, but your evaluation findings will become richer when understood in the broader context that others can provide.

Our thanks to Marianne Philbin, Susie Quern Pratt, and Jenny Ellis Richards.

Leave a comment


Be part of the largest community for funders with few or no staff.

Join Now