Effective Evaluation

FAQs

  • What are some of the greatest pitfalls for funders engaging in evaluation?

  • When should we consider hiring a professional evaluator?

  • What are some of the reasons funders engage in evaluation?

  • How can a funders with few or no staff effectively engage in evaluation?

  • How can we evaluate our foundation’s grantmaking program?

  • How can we engage grantees and grantseekers in assessing our foundation?

Articles

Facilitating Understanding, Communication, and Learning

by Sara Beggs | Sep 10, 2013
As funders work to tackle social problems, they often find themselves as one player among many. All collective efforts, whether formal or informal, can benefit from understanding, communication, and learning—three techniques key to the new field of developmental evaluation. What can small-staff philanthropy learn from this field, and how might you co-opt its most helpful practices to advance your work with others?

Developmental evaluation is a hot topic in philanthropy today. It emerged in response to the need for real-time learning in complex and emerging situations—just the situations in which funders tend to find themselves when working with others to find innovative solutions to social problems.

“Evaluation is about critical thinking; development is about creative thinking,” writes author Jamie Gamble in The J.W. McConnell Family Foundation’s A Developmental Evaluation Primer. “Often these two types of thinking are seen to be mutually exclusive, but developmental evaluation is about holding them in balance.”

In fact, developmental evaluation combines the best of these two very different ways of thinking—both critical for solving problems—and goes beyond what is traditionally considered evaluation to include a host of practices necessary for the often messy work of pursuing change—for example, helping a group to determine a focus, engaging key stakeholders effectively, and facilitating real conversation.

“What [developmental evaluation] does is combine the rigor of evaluation, being evidence-based and objective,” continues Gamble, “with the role of organizational development coaching, which is change-oriented and relational.”

Small-staff philanthropists may find themselves in just the right position—and with just the right skills—to apply key techniques from the field of developmental evaluation to their work with others, allowing for rapid learning, improved strategies, and faster progress toward important goals.

A subset of Exponent Philanthropy’s Impact Working Group joined in a peer learning cohort this year to explore developmental evaluation and its relevance for small-staff philanthropy. Here is some of the group’s early learning.

Three Key Functions



Developmental evaluators don’t tend to have backgrounds in formal evaluation as that term is understood in the nonprofit world. Instead, they are skilled in evaluative thinking, sense-making, and facilitation.

They may serve a project or group in many ways.

  • Keeping in mind the big picture and the small details


  • Asking important questions and stating difficult facts that others may hesitate to mention


  • Understanding the dynamics involved and mediating discussions among key players


  • Helping to build an atmosphere in which funders and grantees share successes and failures


  • Identifying important patterns from field data that individuals may miss


  • Inspiring participants to stay true to larger goals, even when their organizations must compromise


These roles, which may be filled by funders themselves or by consultants they engage, sum to the following actions: facilitating understanding, facilitating communication, and facilitating learning.

Facilitating understanding. We’re all susceptible to being so close to a problem that we lose sight of the factors at play. What a relief when someone with fresh eyes helps to make sense of it. Every messy effort to make change needs someone with this bird’s eye view.

A funder can facilitate understanding and bring a broad perspective to an initiative by:

  • Taking time to learn about key players, root causes, best practices, funding sources, and field dynamics. It is also helpful to summarize the knowledge and test it through further inquiry by asking others to verify if it seems to be on target.


  • Looking for patterns in what is or isn’t working, what is or isn’t funded, or reasons why organizations are or aren’t effective. Funders and grantees focused on their own activities often can’t see, forget to look for, or don’t have time to look for commonalities or trends. Highlighting these patterns can lead to more effective solutions.


  • Promoting understanding. A funder might surface or test assumptions, highlight opportunities for learning, or ask questions that lead the group to think about the implications of its work, think beyond its current activities, or ensure the ultimate goal is still central.


Facilitating communication. Many initiatives flounder for too little communication from the right people, too little communication about the right things, or too much negative communication. This often leads to a lack of trust, a lack of progress, and ultimately a lack of impact.

A funder can facilitate productive communication by:

  • Engaging key stakeholders in the right conversations and at the right level. This involves building awareness of each stakeholder’s critical role, and respecting individual agendas yet motivating the group to build a common actionable agenda.


  • Mediating dynamics by tracking everything from informal conversations to key decisions, understanding each organization’s power to influence and level of engagement, and addressing tensions that will sabotage success. Difficult dynamics are inevitable in any initiative with multiple partners. Being aware of these dynamics while remaining above them is critical.


  • Raising difficult issues. Groups often stagnate without leadership in this area. The person fulfilling this role is often seen as a critical friend—a person on your side and also willing to point out weaknesses. To be effective in this role, a funder must be wholly devoted to the cause, not interested in taking sides, and willing to be the voice of reason that is critical for the initiative’s success.


Facilitating learning. Initiatives with a clear directive to learn are significantly more likely to explore new approaches, admit failures, and learn from missteps. This type of environment is especially important for addressing today’s complex social challenges that lack easy solutions.

Even solutions tried and tested in other communities are likely to require modifications when transplanted to your community.

A funder can facilitate learning by:

  • Setting a learning agenda with stakeholders that is focused on key challenges and opportunities. In many cases, coming to understand a project’s context is a significant piece of this learning. The context is often quite complicated, and initial assumptions about it are often incorrect. Investing in this type of learning can help an initiative to find real and meaningful solutions.


  • Creating incentives for sharing challenges and failures. For grantees, the stakes can’t get any higher than when year-end reports coincide with the possibility of renewed funding. In these situations, grantees are likely to paint a much rosier picture than is reality. Instead, a funder can help to create a learning environment that invites regular sharing of progress, road blocks, and successes—and even praises grantees for their willingness to admit challenges and failures. A learning environment can also encourage funding partners to share and explore the same.



Helpful Skills



Not just anyone will find the previous roles to be a good fit, even if the desire exists. What skills and competencies are key to success? In DE 201: A Practitioner’s Guide to Developmental Evaluation, the authors highlight several capacities:

  • Strategic thinking—Developmental evaluators help others move forward in the face of ambiguity. They are skilled in critical thinking and reasoning, assessing current scenarios, setting priorities, creating logical frameworks, defining actionable goals, and fostering creative solutions.


  • Pattern recognition—Strong analytical skills are necessary for weighing information from complex sources, and extracting themes, trends, and commonalities.


  • Relationship building—Developmental evaluators have excellent people skills. They are able to build unity, address difficult scenarios, ask hard questions, and smooth ruffled feathers. They are highly aware of group dynamics, trusted, and not afraid to say something that is difficult for the group to hear.


  • Servant leadership—Leaders who serve in this way are able to support the group’s work rather than push their own agendas. They are able to celebrate the successes of others and even highlight them ahead of their own.


  • Facilitation skills—Developmental evaluators use strong facilitation skills to create an atmosphere of respect and unity, foster listening among participants, and build shared visions among diverse constituents, ensuring that the voices of key constituents are captured.


To this list, we add:

  • Patience—The road to impact is filled with many twists and turns. Being able to practice patience is critical for success, because the process of facilitating understanding, communication, and learning takes time.


If you are engaged in a project that could benefit from this type of leadership and guidance, consider whether you might fund a consultant to fill some or all of the previous roles. Or, if confident in your abilities, test out a few of the roles to determine how you might contribute to the initiative’s effectiveness moving forward.

What Is Developmental Evaluation?



Michael Quinn Patton’s classic evaluation textbook, Utilization-Focused Evaluation, has served as a kind of Rosetta Stone for a quarter century. Foundations, uncertain how to respond to evaluation needs, have been able to lean on Patton’s text for guidance. Its appeal was its elegance.

“The fundamental premise of the book is that evaluations ought to be useful,” writes Patton in the Fourth Edition. The work “provides an overall framework and concrete advice for how to conduct useful evaluations that actually get used.”

Who could be against useful?

A Different Type of Evaluation



Creating an evaluation department at the Knight Foundation in 1997, I inherited an evaluation of a multicity childhood immunization initiative. As I experienced, even the most well-intentioned evaluations weren’t always useful—particularly in complex situations.

Whereas several trustees had imagined they were funding a straightforward effort to get more vaccines to more kids, most local grantees were starting in a different place. The grantees were in a developmental phase—working to develop good programs—and they had yet to identify all the barriers or troubleshoot all the problems.

The board expected the evaluation to document success—and inform their decisions—by listing numbers of children immunized. But it instead focused—for hundreds of pages—on the barriers to boosting immunization rates. The barriers were top of mind for grantees, and for the staff and evaluators who wished to acknowledge the hard work of these small, community-based organizations.

The evaluation frustrated the board, who anticipated something very different, and neither they nor the grant partners benefited from the process or end result.

Patton, encountering similar situations, decided that what would be most useful would be a version of evaluation that could enhance program and organizational development. So, in 2010, Patton delivered a real gift to the field: Developmental Evaluation, a new evaluation textbook for folks working in messy, complex areas.

Patton explains it this way: “Developmental evaluation is a way of being useful in innovative settings where goals are emergent and changing rather than predetermined and fixed, time periods are fluid and forward-looking rather than artificially imposed by external deadlines, and the purposes are innovation, change and learning rather than external accountability… .”

Applying the Techniques of Developmental Evaluation



Today, having moved from a large foundation to small foundation and from an evaluation role to a program role, I rely on developmental evaluation to support grantees working on tough, ambitious projects.

Our foundation is supporting a health and wellness initiative that aims to help families in a number of ways. Leaders from a team of public and private organizations are working to change a number of systems at once, including public health, public education, food, and human services.

One idea is to use the school as a site for food distribution, providing customized support to meet the needs of individual families. The team has improved the food being served in the school cafeteria, interviewed families to find out how much help they need month to month, and arranged for a mobile pantry to distribute healthy food directly to parents at the school.

A traditional evaluation approach would rely on counts of families served or change in health measures over time. Because this work is new, and it involves changing multiple systems at once, our grant partners are instead focused on testing and developing the best approach.

By seeking feedback from client families and through candid, intense peer learning, they found that two supposed strengths—customization and venue—actually created difficulties.

We believed customizing food distribution would be positive. Instead of foisting upon every family an identical basket of food, our partners attempted to supply families with food to fit their specific situations. Likewise, we viewed the school venue as a convenience for families. Instead of giving parents a referral that directs them to a food pantry in another part of town, our partners were bringing the food pantry to the parents when they were at the school.

In the test, families perceived the customization as judgment, not thoughtfulness. Seeing one family get several bags of food and another family get one bag of food created a sense that arbitrary, unfair decisions had been made. Further, families blamed the school for making the judgments. It was hard for families to separate the venue from the authority behind the program.

For the next school year, our grant partners are considering a variation that would give families more privacy as they pick up their food. The organizations are also questioning whether the school really is the right venue. It is certainly convenient to deliver human services at a school site. But what is the cost of convenience?

Fortunately, developmental evaluation is not about guessing at what someone else wants to know about your program. It is about coming to terms with what you really need to learn to develop an approach that will get you a better result.

John Bare, The Arthur M. Blank Family Foundation

Leave a comment

Samples

Facilitating Understanding, Communication, and Learning

by Sara Beggs | Sep 10, 2013
As funders work to tackle social problems, they often find themselves as one player among many. All collective efforts, whether formal or informal, can benefit from understanding, communication, and learning—three techniques key to the new field of developmental evaluation. What can small-staff philanthropy learn from this field, and how might you co-opt its most helpful practices to advance your work with others?

Developmental evaluation is a hot topic in philanthropy today. It emerged in response to the need for real-time learning in complex and emerging situations—just the situations in which funders tend to find themselves when working with others to find innovative solutions to social problems.

“Evaluation is about critical thinking; development is about creative thinking,” writes author Jamie Gamble in The J.W. McConnell Family Foundation’s A Developmental Evaluation Primer. “Often these two types of thinking are seen to be mutually exclusive, but developmental evaluation is about holding them in balance.”

In fact, developmental evaluation combines the best of these two very different ways of thinking—both critical for solving problems—and goes beyond what is traditionally considered evaluation to include a host of practices necessary for the often messy work of pursuing change—for example, helping a group to determine a focus, engaging key stakeholders effectively, and facilitating real conversation.

“What [developmental evaluation] does is combine the rigor of evaluation, being evidence-based and objective,” continues Gamble, “with the role of organizational development coaching, which is change-oriented and relational.”

Small-staff philanthropists may find themselves in just the right position—and with just the right skills—to apply key techniques from the field of developmental evaluation to their work with others, allowing for rapid learning, improved strategies, and faster progress toward important goals.

A subset of Exponent Philanthropy’s Impact Working Group joined in a peer learning cohort this year to explore developmental evaluation and its relevance for small-staff philanthropy. Here is some of the group’s early learning.

Three Key Functions



Developmental evaluators don’t tend to have backgrounds in formal evaluation as that term is understood in the nonprofit world. Instead, they are skilled in evaluative thinking, sense-making, and facilitation.

They may serve a project or group in many ways.

  • Keeping in mind the big picture and the small details


  • Asking important questions and stating difficult facts that others may hesitate to mention


  • Understanding the dynamics involved and mediating discussions among key players


  • Helping to build an atmosphere in which funders and grantees share successes and failures


  • Identifying important patterns from field data that individuals may miss


  • Inspiring participants to stay true to larger goals, even when their organizations must compromise


These roles, which may be filled by funders themselves or by consultants they engage, sum to the following actions: facilitating understanding, facilitating communication, and facilitating learning.

Facilitating understanding. We’re all susceptible to being so close to a problem that we lose sight of the factors at play. What a relief when someone with fresh eyes helps to make sense of it. Every messy effort to make change needs someone with this bird’s eye view.

A funder can facilitate understanding and bring a broad perspective to an initiative by:

  • Taking time to learn about key players, root causes, best practices, funding sources, and field dynamics. It is also helpful to summarize the knowledge and test it through further inquiry by asking others to verify if it seems to be on target.


  • Looking for patterns in what is or isn’t working, what is or isn’t funded, or reasons why organizations are or aren’t effective. Funders and grantees focused on their own activities often can’t see, forget to look for, or don’t have time to look for commonalities or trends. Highlighting these patterns can lead to more effective solutions.


  • Promoting understanding. A funder might surface or test assumptions, highlight opportunities for learning, or ask questions that lead the group to think about the implications of its work, think beyond its current activities, or ensure the ultimate goal is still central.


Facilitating communication. Many initiatives flounder for too little communication from the right people, too little communication about the right things, or too much negative communication. This often leads to a lack of trust, a lack of progress, and ultimately a lack of impact.

A funder can facilitate productive communication by:

  • Engaging key stakeholders in the right conversations and at the right level. This involves building awareness of each stakeholder’s critical role, and respecting individual agendas yet motivating the group to build a common actionable agenda.


  • Mediating dynamics by tracking everything from informal conversations to key decisions, understanding each organization’s power to influence and level of engagement, and addressing tensions that will sabotage success. Difficult dynamics are inevitable in any initiative with multiple partners. Being aware of these dynamics while remaining above them is critical.


  • Raising difficult issues. Groups often stagnate without leadership in this area. The person fulfilling this role is often seen as a critical friend—a person on your side and also willing to point out weaknesses. To be effective in this role, a funder must be wholly devoted to the cause, not interested in taking sides, and willing to be the voice of reason that is critical for the initiative’s success.


Facilitating learning. Initiatives with a clear directive to learn are significantly more likely to explore new approaches, admit failures, and learn from missteps. This type of environment is especially important for addressing today’s complex social challenges that lack easy solutions.

Even solutions tried and tested in other communities are likely to require modifications when transplanted to your community.

A funder can facilitate learning by:

  • Setting a learning agenda with stakeholders that is focused on key challenges and opportunities. In many cases, coming to understand a project’s context is a significant piece of this learning. The context is often quite complicated, and initial assumptions about it are often incorrect. Investing in this type of learning can help an initiative to find real and meaningful solutions.


  • Creating incentives for sharing challenges and failures. For grantees, the stakes can’t get any higher than when year-end reports coincide with the possibility of renewed funding. In these situations, grantees are likely to paint a much rosier picture than is reality. Instead, a funder can help to create a learning environment that invites regular sharing of progress, road blocks, and successes—and even praises grantees for their willingness to admit challenges and failures. A learning environment can also encourage funding partners to share and explore the same.



Helpful Skills



Not just anyone will find the previous roles to be a good fit, even if the desire exists. What skills and competencies are key to success? In DE 201: A Practitioner’s Guide to Developmental Evaluation, the authors highlight several capacities:

  • Strategic thinking—Developmental evaluators help others move forward in the face of ambiguity. They are skilled in critical thinking and reasoning, assessing current scenarios, setting priorities, creating logical frameworks, defining actionable goals, and fostering creative solutions.


  • Pattern recognition—Strong analytical skills are necessary for weighing information from complex sources, and extracting themes, trends, and commonalities.


  • Relationship building—Developmental evaluators have excellent people skills. They are able to build unity, address difficult scenarios, ask hard questions, and smooth ruffled feathers. They are highly aware of group dynamics, trusted, and not afraid to say something that is difficult for the group to hear.


  • Servant leadership—Leaders who serve in this way are able to support the group’s work rather than push their own agendas. They are able to celebrate the successes of others and even highlight them ahead of their own.


  • Facilitation skills—Developmental evaluators use strong facilitation skills to create an atmosphere of respect and unity, foster listening among participants, and build shared visions among diverse constituents, ensuring that the voices of key constituents are captured.


To this list, we add:

  • Patience—The road to impact is filled with many twists and turns. Being able to practice patience is critical for success, because the process of facilitating understanding, communication, and learning takes time.


If you are engaged in a project that could benefit from this type of leadership and guidance, consider whether you might fund a consultant to fill some or all of the previous roles. Or, if confident in your abilities, test out a few of the roles to determine how you might contribute to the initiative’s effectiveness moving forward.

What Is Developmental Evaluation?



Michael Quinn Patton’s classic evaluation textbook, Utilization-Focused Evaluation, has served as a kind of Rosetta Stone for a quarter century. Foundations, uncertain how to respond to evaluation needs, have been able to lean on Patton’s text for guidance. Its appeal was its elegance.

“The fundamental premise of the book is that evaluations ought to be useful,” writes Patton in the Fourth Edition. The work “provides an overall framework and concrete advice for how to conduct useful evaluations that actually get used.”

Who could be against useful?

A Different Type of Evaluation



Creating an evaluation department at the Knight Foundation in 1997, I inherited an evaluation of a multicity childhood immunization initiative. As I experienced, even the most well-intentioned evaluations weren’t always useful—particularly in complex situations.

Whereas several trustees had imagined they were funding a straightforward effort to get more vaccines to more kids, most local grantees were starting in a different place. The grantees were in a developmental phase—working to develop good programs—and they had yet to identify all the barriers or troubleshoot all the problems.

The board expected the evaluation to document success—and inform their decisions—by listing numbers of children immunized. But it instead focused—for hundreds of pages—on the barriers to boosting immunization rates. The barriers were top of mind for grantees, and for the staff and evaluators who wished to acknowledge the hard work of these small, community-based organizations.

The evaluation frustrated the board, who anticipated something very different, and neither they nor the grant partners benefited from the process or end result.

Patton, encountering similar situations, decided that what would be most useful would be a version of evaluation that could enhance program and organizational development. So, in 2010, Patton delivered a real gift to the field: Developmental Evaluation, a new evaluation textbook for folks working in messy, complex areas.

Patton explains it this way: “Developmental evaluation is a way of being useful in innovative settings where goals are emergent and changing rather than predetermined and fixed, time periods are fluid and forward-looking rather than artificially imposed by external deadlines, and the purposes are innovation, change and learning rather than external accountability… .”

Applying the Techniques of Developmental Evaluation



Today, having moved from a large foundation to small foundation and from an evaluation role to a program role, I rely on developmental evaluation to support grantees working on tough, ambitious projects.

Our foundation is supporting a health and wellness initiative that aims to help families in a number of ways. Leaders from a team of public and private organizations are working to change a number of systems at once, including public health, public education, food, and human services.

One idea is to use the school as a site for food distribution, providing customized support to meet the needs of individual families. The team has improved the food being served in the school cafeteria, interviewed families to find out how much help they need month to month, and arranged for a mobile pantry to distribute healthy food directly to parents at the school.

A traditional evaluation approach would rely on counts of families served or change in health measures over time. Because this work is new, and it involves changing multiple systems at once, our grant partners are instead focused on testing and developing the best approach.

By seeking feedback from client families and through candid, intense peer learning, they found that two supposed strengths—customization and venue—actually created difficulties.

We believed customizing food distribution would be positive. Instead of foisting upon every family an identical basket of food, our partners attempted to supply families with food to fit their specific situations. Likewise, we viewed the school venue as a convenience for families. Instead of giving parents a referral that directs them to a food pantry in another part of town, our partners were bringing the food pantry to the parents when they were at the school.

In the test, families perceived the customization as judgment, not thoughtfulness. Seeing one family get several bags of food and another family get one bag of food created a sense that arbitrary, unfair decisions had been made. Further, families blamed the school for making the judgments. It was hard for families to separate the venue from the authority behind the program.

For the next school year, our grant partners are considering a variation that would give families more privacy as they pick up their food. The organizations are also questioning whether the school really is the right venue. It is certainly convenient to deliver human services at a school site. But what is the cost of convenience?

Fortunately, developmental evaluation is not about guessing at what someone else wants to know about your program. It is about coming to terms with what you really need to learn to develop an approach that will get you a better result.

John Bare, The Arthur M. Blank Family Foundation

Leave a comment

Additional Resources

MEMBERSHIP

Be part of the largest community for funders with few or no staff.

Join Now