|
USAID continues to improve its evaluation efforts, including in the democracy, human rights, and governance (DRG) sector. Nonetheless, despite a well-institutionalized evaluation policy, we need to be doing more rigorous evaluations, including more impact evaluations (IEs) AND more rigorous performance evaluations (PEs) that focus on outcomes and include collecting baseline and endline data. Furthermore, evaluations don’t end with the completion of the report, and USAID operating units (OUs) need to be developing required post-evaluation action plans and spending more time using evaluation findings. These needs are highlighted in a recent Vox article.
 |
For the last decade, the USAID DRG Center’s Evidence and Learning (E&L) Team has been working with the global DRG cadre and implementing partners to improve the quality and use of DRG sector evaluations. Despite progress in this regard, four recent studies profiled in this DRG Learning Digest illustrate some ongoing challenges and limitations with USAID DRG evaluations.
Specifically, in this edition we will explore how:
- IEs and rigorous PEs focused on outcomes are still relatively rare, and when we do undertake them we struggle with their implementation.
- There is considerable room for improvement in the utilization of evaluation findings.
- The DRG Center is responding to these findings by encouraging more rigorous evaluations and better utilization.
Please make use of DRG Evidence and Learning Team resources! (See text box at the end.)
Evaluation Planning: Impact Evaluations and the “Missing Middle”
USAID is conducting many evaluations in the DRG sector, but could be doing much more, in particular regarding the quality of those evaluations. According to the USAID Evaluation Registry, from 2016 to 2020 the Agency completed 173 external DRG evaluations valued at over $47 million (see Table 1). On the one hand, this is an impressive quantity, and this investment outpaces many other donors. On the other hand, it represents less than one percent of DRG program expenditures over this period, far less than the previous Agency target of three percent, which was more recently lowered in ADS 201.3.6.5 to one to three percent.
 Furthermore, this table shows that USAID DRG IEs are still relatively rare. We used to think that it was too difficult to do IEs in the DRG sector. However, over the past 15 years USAID and many other organizations—such as the Abdul Latif Jameel Poverty Action Lab (J-PAL) and Evidence in Governance and Politics (EGAP)—have disproved that assumption by completing many successful DRG IEs. In fact, the International Initiative for Impact Evaluation (3ie) has identified 1,625 DRG IEs compiled into six evidence maps.
While the Evaluation Registry does not provide a breakdown of PE types, we know that the vast majority of these are one-time PEs conducted sometime between the midpoint and endpoint of an activity. These typically entail three weeks of field work and a mix of key informant interviews and focus group discussions with beneficiaries and other stakeholders. Often there is no baseline data to inform these PEs. While these evaluations can serve as a learning resource, they are not well equipped to measure outcomes or changes in outcomes over time or to detect program contributions to these changes. In fact, in the DRG sector there appear to be very few instances outside of IEs in which an external evaluation team measures outcomes of an activity over time. (See tips on such evaluations in the DRG sector.)
Figure 1: The “missing middle” in performance evaluation rigor.
In short, there is a “missing middle” in evaluation implementation, with a small number of rigorous IEs, a large number of one-time PEs of varying degrees of rigor, and a missing middle in between (see Figure 1). This is not because Missions do not want to understand outcomes and impacts. An unpublished study by the USAID Bureau for Policy, Planning, and Learning (PPL) of a sample of 100 PEs found that close to one-third included questions that could only be answered by an IE.
A forthcoming study on the role of evaluation planning in the DRG activity design process suggests that the lack of IEs and rigorous PEs is a result of several factors. First, the one-shot PE is heavily ingrained and is simply a default option. A textual analysis of USAID DRG Requests for Proposals (RFPs) between 2018 and 2020 found that when there were references to evaluations, Missions tend to use stock language consistent with this style of evaluation. Second, until revisions in 2020, the ADS did not distinguish between types of PEs, and the one-shot PE still meets the Agency’s evaluation requirements. Third, more rigorous evaluations cost more and in the case of IEs require considerable effort to plan and implement. Finally, the activity design process involves a number of challenges and bureaucratic hoops that relegate evaluation planning to a low priority.
The E&L Team completed a recent retrospective of 27 USAID DRG IEs conducted since 2010 and found considerable implementation challenges. In addition to challenges outside of USAID’s control (e.g., changes in the political or security situation), many of these IEs were undermined by disagreements and tension between evaluators and implementers. Implementing partners (IPs) were frustrated at the constraints the evaluation design placed on implementation and evaluators were frustrated with IP defensiveness and perceived resistance to the evaluation due to reputational risk. A broader USAID study also raised serious concerns with IE quality.
In short, despite impressive institutionalization of evaluation within the DRG sector, we are not doing all the ADS requires, IEs remain relatively rare, there is a large missing middle, and there are challenges to getting all of these kinds of evaluations right.
Utilization: Moving Beyond Just a Deliverable
The second major challenge is about what happens after the evaluation is completed. One of the main goals of evaluation is to use learning to adapt; however, producing the deliverable of an evaluation report often becomes an end in itself rather than a means to better programming. The USAID DRG IE retrospective report cited above did find some instances in which IEs had significant influence (e.g., an IE in Haiti which led to pre-trial detention reforms); however, the study generally found low levels of utilization. This was attributed to evaluations completed after decisions needed to be made, user-unfriendly reports, limited dissemination and readership, and a lack of post-evaluation action plans.
In addition to the retrospective, the E&L Team commissioned its learning partners to follow up with Missions and other OUs on the utilization of past evaluations and other studies. Like the retrospective, the NORC-Social Impact and Cloudburst studies also found some examples of significant utilization (e.g., a series of evidence reviews that were successfully used to help defend a procurement decision under Congressional scrutiny); however, study participants generally had a hard time articulating clear examples of how evaluations were used. From the launch of the USAID Evaluation Policy in 2011 until 2016, there was an enormous focus on improving evaluations but little to no focus on what happens after the evaluation is completed. It wasn’t until 2016 that the ADS was revised to require OUs to complete a post-evaluation action plan. However, as this was after the big evaluation push in the Agency, the studies and DRG Center experience suggest that awareness of and compliance with this requirement is low. Of 26 evaluations examined in the studies, respondents only reported six action plans, and of 47 learning, evaluation, and research initiatives only one could produce an updated action plan monitoring implementation progress.
The studies enumerated a wide range of barriers to utilization, among others including (1) a tendency to view evaluations as a “check-the-box” requirement, (2) poor incentives that don’t encourage utilization, (3) a tendency to view evaluations as audits rather than learning opportunities, (4) staff turnover, (5) a perception that the evaluation did not provide new information, (6) resource constraints, (7) poor dissemination, and (8) timing problems. Regarding the second point, the report cites one interviewee as saying:
“Incentives are very positive for staff to invest in and commit to evaluations, but the incentives drop off when it comes to actually following up and implementing those recommendations and responding to those findings.”
DRG Center Response: Fostering Rigorous Evaluations in DRG (FRED) and Encouraging Utilization
Given these twin challenges, the DRG Center has been pursuing two new initiatives.
First, the E&L Team continues to support IEs, including through both technical support and co-funding. The retrospective study recommended such support as essential, given the technical nature of IEs and the difficulty for Missions in covering the full cost. We call this initiative Fostering Rigorous Evaluations in DRG or FRED (link only accessible to USAID personnel).
To support this initiative, the Center is taking a number of steps to address the main challenge identified by the retrospective: promoting greater collaboration between evaluators and implementers. Most important of these is for each evaluation to organize a co-design workshop involving the implementer, its partners, and host country government and/or other local stakeholders, depending on the nature of the activity. Other changes to the evaluation design process include: (1) commissioning a formal evidence review to ensure the selected programmatic approach has an adequately robust theory of change; (2) conducting an evaluability assessment to inform the workshop; (3) developing an annex to the evaluation design report that lays out stakeholder commitments; (4) improving knowledge management and stakeholder communication; and (5) openness to IP participation on the evaluation team. This new evaluation design and implementation process, which includes multiple decision points at which a Mission can decide to move forward or not, is illustrated below.
 Recognizing the problem of the missing middle, DRG Center support to Missions is not limited to IEs, but also rigorous, outcome-oriented PEs. These are to include at least a baseline and endline measure of outcomes and efforts to understand how the intervention may or may not have contributed to any observed change.
Second, the DRG Center is in the nascent phase of implementing a new process to promote and track utilization across all research efforts under the DRG Learning, Evaluation, and Research (LER) II and subsequent mechanisms. This includes the following steps:
- Learning partners draft an action plan template that includes summaries of the recommendations for the OU to respond to.
- Learning partners convene a utilization workshop as the evaluation concludes, to help prioritize among recommendations and potential action items.
- About three months later, the E&L Team and its learning partners follow up to explore action plan implementation and evaluation utilization. This process is then repeated typically around six months after evaluation completion.
- These efforts are tracked to monitor a series of indicators (e.g., extent of implementation and updating of an action plan).
- The E&L Team is also working on developing a short one-hour briefing for Mission staff on evaluation utilization.
Further Assistance
To learn more about promoting rigorous evaluations and fostering norms of evaluation utilization, please contact the DRG Center’s Evidence and Learning Team at ddi.drg.elmaillist@usaid.gov or Daniel Sabet at dsabet@usaid.gov.
USE OUR RESOURCES!
Welcome to the DRG Learning Digest, a newsletter to keep you informed of the latest learning, evaluation, and research in the Democracy, Human Rights, and Governance (DRG) sector. Views expressed in the external (non-USAID) publications linked in this Digest do not necessarily represent the views of the United States Agency for International Development or the United States Government.
Don't forget to check out our DRG Learning Menu of Services! (Link only accessible to USAID personnel.) The Menu provides information on the learning products and services the Evidence and Learning Team offers to help you fulfill your DRG learning needs. We want to help you adopt learning approaches that emphasize best fit and quality.
The Evidence and Learning Team is also excited to share our DRG Learning, Evidence, and Analysis Platform (LEAP) with you. This Platform contains an inventory of programmatic approaches, evidence gap maps, the DRG Learning Harvest, and inventories of indicators and country data portraits - all of which can be very useful in DRG activity design, implementation, evaluation, and adaptation. Some of these resources are still being built, so check back frequently to see what has been newly added.
The DRG Learning Harvest on LEAP is a searchable database of DRG learning products, including summaries of key findings and recommendations, drop-down menus to easily find documents related to a particular country or program area, and links to the full reports on the DEC.
Our friends at the Varieties of Democracy (V-Dem) Institute are also seeking to expand their research partnership with USAID on the complex nature of democracy by inviting research questions from you for V-Dem to work on. If there's a DRG technical question you've been wondering about, please email the Evidence and Learning Team at ddi.drg.elmaillist@usaid.gov.
|
