State of Evaluation 2012: Evaluation Practice and Capacity in the Nonprofit Sector
2 likes2,469 views
The document presents the findings from the 2012 State of Evaluation survey, focusing on the evaluation practices and capacity among U.S.-based 501(c)(3) nonprofit organizations. With a response rate of 4%, the survey highlights various aspects including evaluation purposes, methodologies, staffing, and barriers faced by nonprofits. It also emphasizes the role of data visualization in enhancing the understanding of evaluation results.
State of Evaluation 2012: Evaluation Practice and Capacity in the Nonprofit Sector
- 1. state of evaluation 2012 EVALUATION PRACTICE AND CAPACITY IN THE NONPROFIT SECTOR Johanna Morariu, Kat Athanasiades, & Ann Emery [email protected] Washington Evaluators Brown Bag Washington, DC February 25, 2013
- 2. www.stateofevaluation.org | Page 2 | #soe2012 Do you have experience in the nonprofit or philanthropic fields? Tell me more! Based on what you know or may have heard about nonprofit evaluation, how do you think nonprofits are doing? Have you heard about data viz (visualization)? What do you think about the role of data viz in evaluation? Welcome!
- 3. www.stateofevaluation.org | Page 3 | #soe2012 Background
- 4. www.stateofevaluation.org | Page 4 | #soe2012 Population of interest: U.S.-based 501(c)3 nonprofit organizations Using GuideStar data, the sampling frame was 501(c)3 Public Charities that had updated their IRS Form 990 in 2010 or more recently, and had provided a contact name and email address A total of 38,789 organizations met the criteria, and a random sample of 20,000 organizations were invited to participate in the survey The survey was available online three weeks in August 2012 and received 567 responses from representatives of nonprofit organizations The survey response rate calculated on complete responses was 4% Methodology
- 5. www.stateofevaluation.org | Page 5 | #soe2012 Comparison of Budget by Sample
- 6. www.stateofevaluation.org | Page 6 | #soe2012 Geographic Distribution of Respondents
- 7. www.stateofevaluation.org | Page 7 | #soe2012 Role of Respondents
- 8. www.stateofevaluation.org | Page 8 | #soe2012 Organization Age
- 9. www.stateofevaluation.org | Page 9 | #soe2012 Data Visualization
- 10. 1 Overview of Results Purpose, Approaches, & Resourcing Evaluation Practice Staffing for Evaluation Audience & Use Barriers to Evaluation 2 3 4 5 6
- 11. 1 Overview of Results Purpose, Approaches, & Resourcing Evaluation Practice Staffing for Evaluation Audience & Use Barriers to Evaluation 2 3 4 5 6
- 12. Overview of Results www.stateofevaluation.org | Page 12 | #soe2012
- 13. Overview of Results www.stateofevaluation.org | Page 13 | #soe2012
- 14. www.stateofevaluation.org | Page 14 | #soe2012 Overview of Results
- 15. 1 Overview of Results Purpose, Approaches, & Resourcing Evaluation Practice Staffing for Evaluation Audience & Use Barriers to Evaluation 2 3 4 5 6
- 16. Purpose, Approaches, & Resourcing www.stateofevaluation.org | Page 16 | #soe2012
- 17. Purpose, Approaches, & Resourcing www.stateofevaluation.org | Page 17 | #soe2012
- 18. www.stateofevaluation.org | Page 18 | #soe2012 Purpose, Approaches, & Resourcing
- 19. 1 Overview of Results Purpose, Approaches, & Resourcing Evaluation Practice Staffing for Evaluation Audience & Use Barriers to Evaluation 2 3 4 5 6
- 20. www.stateofevaluation.org | Page 20 | #soe2012 Evaluation Practice
- 21. www.stateofevaluation.org | Page 21 | #soe2012 Evaluation Practice
- 22. www.stateofevaluation.org | Page 22 | #soe2012 Evaluation Practice
- 23. 1 Overview of Results Purpose, Approaches, & Resourcing Evaluation Practice Staffing for Evaluation Audience & Use Barriers to Evaluation 2 3 4 5 6
- 24. www.stateofevaluation.org | Page 24 | #soe2012 Staffing for Evaluation
- 25. www.stateofevaluation.org | Page 25 | #soe2012 Staffing for Evaluation
- 26. www.stateofevaluation.org | Page 26 | #soe2012 Staffing for Evaluation
- 27. 1 Overview of Results Purpose, Approaches, & Resourcing Evaluation Practice Staffing for Evaluation Audience & Use Barriers to Evaluation 2 3 4 5 6
- 28. www.stateofevaluation.org | Page 28 | #soe2012 Audience & Use
- 29. Audience & Use www.stateofevaluation.org | Page 29 | #soe2012
- 30. www.stateofevaluation.org | Page 30 | #soe2012 Audience & Use
- 31. 1 Overview of Results Purpose, Approaches, & Resourcing Evaluation Practice Staffing for Evaluation Audience & Use Barriers to Evaluation 2 3 4 5 6
- 32. www.stateofevaluation.org | Page 32 | #soe2012 Barriers to Evaluation
- 33. www.stateofevaluation.org | Page 33 | #soe2012 Barriers to Evaluation
- 34. Related Reports Other reports exist that measure different aspects of evaluation in the social sector. If you’d like to take a look at what other organizations are finding, check out these resources:
- 35. Other Resources There are many, many more papers, interviews, and tools available— for free—through Innovation Network’s Point K Learning Center. Check it out! Here’s where you can find the Point K Learning Center on our home page, www.innonet.org:
- 36. Discussion Questions 1 How do these findings fit with your professional experience? Which findings do you find most interesting? Are there questions you suggest we consider including next time? How might these findings impact your work? In your opinion, did the data visualization enhance or detract from the presentation of findings? What are the biggest takeaway messages for you from this work? 2 3 4 5 6 Based on what you know about nonprofit evaluation, how do you think nonprofits are doing?7
- 37. www.stateofevaluation.org #soe2012 Thank You! Johanna Morariu, Kat Athanasiades, & Ann Emery [email protected] @InnoNet_Eval
Editor's Notes
- #2: JM
- #4: JM
- #5: JM
- #6: KAWe compared our sample with the Nonprofit Sector in Brief 2011 Public Charity Statistics. The Public Charity Statistics are the periwinkle/light blue colors, while our survey respondents are in the darker blue.These buckets are based on the organization’s budget size.As you can see, a number of the buckets, while not the same, track relatively closely. However, there is divergence between our sample and national statistics in terms of the smallest nonprofits and those nonprofits with budgets from $1 to $4.99 million. We could imagine reasons for this, but we don’t know why there is this discrepancy, so it is good to remember that some organizations are overrepresented by our sample and some are underrepresented.This is somewhat mitigated on the high end by how we split the data—in the State of Evaluation, we often compare what we define as “smaller” organizations (those with annual budgets under $500,000) and those that are large (annual budgets over $5 million). Our “large” organizations, then, the last two buckets on the right, track closely to the national statistics.
- #7: KAWe also analyzed the geographic distribution of our survey respondents. As you can see, 31% were from the West (including AK and HI), 18% were from the Midwest, 32% were from the South, and 19% were from the Northeast. So all states (IS THIS TRUE? If not, can say “regions”) are represented in this survey.[Q: DO WE HAVE A BREAKDOWN OF THE NATIONAL STATISTICS TO COMPARE TO OUR SAMPLE?]
- #8: AE
- #9: AE
- #10: AETop left: screen grab of some SOE dataBottom left: Organizing data into findingsRight: Visualizing findings in Word. These mockups are what we sent to the graphic designer (and we will see how closely they track to what the graphic designer created).
- #11: JM
- #12: JM
- #13: JM
- #14: JM
- #15: JM
- #16: KAIn the next section of the State of Evaluation, we looked at Purpose of evaluation, Approaches to evaluation, and Resourcing for evaluation.
- #17: KAWe asked the question, “When evaluating your work, how important are the following?”Those response choices most viewed as very important or somewhat important were:“Strengthen future work,”“Learn whether original objectives were achieved,” and“Learn about outcomes.”The least important was“Strengthen public policy,” followed by“Contribute to knowledge in the field.”We do a lot of advocacy work, and we also try to assemble evaluation knowledge and put it out in the field, so these last two columns are very important to us, though it is logical that nonprofits want to use evaluation to inform their work going forward.
- #18: KAWe asked about what evaluation approaches organizations used in the past year. Organizations could select more than one, and we explained that it wasn’t likely that organizations employed all of these approaches.As you can see, of those that evaluated their work, 79% said that they engaged in outcomes evaluation, 65% in before and after measures, and 48% in impact evaluation. The least used methods of evaluation are quasi experimental research, RCTs, and studies with control groups.We think that this information really provides insight into what organizations consider evaluation. While looking at these results, we did question whether 79% of these organizations really did engage in outcomes evaluation as it is used in the field. We’ve often seen confusion between outputs and outcomes, for example, in estimating effectiveness. It was also enlightening to look at the open-ended responses to “How does your organization define evaluation?”So we understand there might be some methodological issues here with reliability of responses, but it is still heartening that 90% of organizations evaluate their projects in some way, and it is useful to know that Outcomes evaluation seems to be trending right now among nonprofits.
- #19: KAThis is a finding that has drawn some criticism, though it is also a useful metric to explore. We found that more than 70% of organizations are spending less than 5% on evaluation. Is 5% an arbitrary amount? The benchmark of 5% is more conservative than the CDC’s benchmark of [X%]. We recommend that an organization try to allocate 5% of their budget to evaluation, whether expending that on an internal evaluator, an external consultant, or a mix.
- #20: AE
- #21: AE
- #22: AE
- #23: AE
- #24: JM
- #25: JM
- #26: JM
- #27: JM
- #28: KAWho is typically the Audience for evaluation, and how is evaluation Used by nonprofits?
- #29: KAWell, we asked survey respondents to rank these three questions by importance to their organization. And we were thrilled to see these results. The primary concern for organizations is “What difference did [our intervention] make?” This corresponds to a question about whether a nonprofit saw success achieving impact.“How well did we do?” came in 2nd in terms of importance, and really tells you whether a nonprofit is successful in achieving their outcomes.And #3, “How much did we do?” corresponds to what outputs an organization achieved.
- #30: KAThis and the next were some of our favorite visualizations in the State of Evaluation, both because we (or I) liked how they looked, as well as because of the story they tell. 100% of organizations used their evaluation findings! That means that nonprofits use their findings to update their Boards, to report to funders, to revise programs, and in a smaller percentage, to implement scaling strategies. No organizations wasted their evaluation findings.
- #31: KAAnd 100% of organizations communicated their findings—similarly to the previous slide, these were often communicated in updates to Boards or reports to funders, but often as well in staff meetings. Buy in among staff is such an important factor to incorporating evaluation in programs, so this is a step.
- #32: AE
- #33: AE
- #34: AE
- #35: JMOriginal slide text:Other reports exist that measure different aspects of evaluation in the social sector. If you’d like to take a look at what other organizations are finding, check out these resources:Center for Effective Philanthropy (CEP). Room for Improvement: Foundations’ Support of Nonprofit Performance Assessment. September 2012.New Philanthropy Capital (NPC). Making an Impact: Impact measurement among charities and social enterprises in the UK. October 2012.Innovation Network. State of Evaluation 2010: Nonprofit Evaluation and Capacity. October 2010.
- #36: JM
- #37: KA & AE
- #38: KA & AEThanks, and that brings us to the end of our presentation! If you’d like to read more or download the State of Evaluation, you can visit stateofevaluation.org, and you can tweet with the hashtag #soe2012. We encourage you to take a closer look at the information presented in the SOE and let us know what you think—your interpretations of the data, your questions, and your feedback more generally.