How to establish a Useful Evaluation Function: It is a Balancing Act
By Caroline Heider, Director General of the Independent Evaluation Group at the World Bank (IEG)
Today, as I reflect on what makes for a useful evaluation function, I remain convinced of the critical importance of these four elements - utility, impartiality, credibility and outreach.
Ten years ago, Ray Rist, the then president of the International Development Evaluation Association, invited me to be a keynote speaker at IDEAS' biannual conference.* The conference's topic was evaluation capacity development. At the time, I had already dedicated over twenty years of my career to making evaluation work, which meant for me enhancing its utility by strengthening impartiality, credibility, and outreach. In addition, I had evaluated capacity development projects, which gave me insights into concepts and processes that were applicable to developing evaluation capacities as well.
Today, as I reflect on what makes for a useful evaluation function, I remain convinced of the critical importance of these four elements - utility, impartiality, credibility and outreach.
A useful evaluation function also requires multiple layers—from enabling environment to institutionalized systems and individual aptitudes and skills—and involves an iterative process with a long-term vision, adaptable tactics, and continuous course-corrections.
Enabling Environment
The concept of an enabling environment dates to the origins of the Independent Evaluation Group in July 1973. In the early days, a mandate (or policy, as it is often called in other institutions) was necessary to specify roles and responsibilities. Already 45 years ago, the World Bank had understood the importance of an independent voice with authority to speak truth to power, assess success and failure, and recommend ways for improvements.
Today, I would argue that such policy-backing continues to be necessary—evaluation offices typically come under pressure when reporting evidence that contradicts the norms or existing strategic directions but it also needs to go hand-in-hand with a culture that embraces learning and accountability.
In such a culture, each of the parties needs to contribute its share:
- Evaluation must work to the highest professional standards and with a clear intent to help improve results and performance,
- Management should create an atmosphere of accountability rather than blame to ensure responsible conduct, learning and timeline course-corrections, and the
- Oversight body—such as executive boards or parliaments—ought to hold the institution or government accountable and incentivize learning.
Institutionalized Systems
Independence has often been interpreted as the evaluators' freedom to do whatever they want or as means to/reason for isolation. By contrast, in my practice independence is a means to impartiality. By removing influence—through decision-powers, budget, or human resource decisions—and combined with the commitment to highest possible standards, evaluation can aspire to impartiality.**
Why is that important? Because an unbiased view provides balanced evidence on the effects of policy-choices or ways of program implementation; information that can help decision-makers and operational teams make better informed choices.
Such aspiration cannot be achieved just by personal commitment, but also requires establishing systems:
- For the choice of evaluations to be strategically relevant on the one hand, and not leave material gaps in the coverage of the institution's work on the other. Often, this dual ambition requires using different types of evaluations for different purposes. This may include project-level evaluations to achieve representative coverage, major evaluations that are strategically selected. Selectivity systems can include analytics, gap analyses, consultation processes, mapping cutting issues and need/opportunity for evaluation so that choices are based on a set of principles and a system that renders the best choices.
- For quality and credibility, the evaluation function needs to establish standards and quality assurance processes. Both need to be tailored to the circumstances of the function and institution, whereby in some situation principle-based standards are more effective and in others somewhat more prescriptive guidelines are needed. Protocols for data collection and analyses are essential to minimize inadvertent biases that could occur otherwise.
- In addition, evaluation processes must comprise adequate consultation with stakeholders, including program managers. This is necessary to ensure transparency of the process, increase trust and credibility, and provide opportunities for learning during evaluation. Institutionalizing such processes, in the sense of establishing norms or agreements, helps create greater reliability (rather than dependence on personal preferences) and safeguards the evaluation team from undue pressure.
- Finally, arrangements for outreach and communications benefit greatly from an established system that assesses opportunities and weighs different options to ensure evaluation findings and messages reach the right audiences at the right time. Without such a systematic approach, outreach can be patchy, inefficient, and not particularly strategic.
Individual Aptitudes and Skills
The job of an evaluator is hard! One has to be analytically meaningful, technically sound, humanly engaging, a strategic thinker, and an excellent communicator. Or, at least that would be the super-hero model...
Among the many advantages of IEG, we have been able to put together an academy that combines regular courses of the World Bank Group that are relevant to our work with specific evaluation training, communication workshops and others to offer continuous learning opportunities to our staff. This is the exception and not the norm. But, any evaluation function should invest in its people to perform at their best.
Less the responsibility of the evaluation function, but equally important, are decision-makers and program managers who are well versed in understanding evaluation—educated users, who recognize the value of evaluation and know how to use findings, lessons, and recommendations for better results and performance.
How then do these building blocks—enabling environment, institutionalized systems, and individual skills—come together? ... it's an iterative process
Developing a function that meets these requirements is not easy. Above all, it is not something one can plan with a blueprint and implement. Rather, it takes a process of continuous diagnostics—testing and implementation—assessment—continuous learning and adjustment as well as jumping on opportunities when they areise.
*Following my presentation, Ray Rist also invited me to contribute to IEG's publication Influencing Change.
**I have no illusions about human imperfection, including subjectivity, and that ultimate impartiality is unattainable. But, I also believe that as evaluator we need to hold ourselves to such standard.
