1. Background
The Department for International Development (DfiD) in collaboration with the UK Evaluation Society (UKES) offered bursaries to members of Evaluation Associations in developing countries to attend the UKES annual conference, covering travel, accommodation and subsistence. The conference took place on November 24th and 25th in London. The South African Monitoring and Evaluation Association nominated 5 candidates with the highest merit according to DfID requirements, and as a result of the nomination, the DfiD and UKES awarded a bursary to three SAMEA applicants, namely Candice Morkel, Senior Manager: M&E in the EC Office of the Premier, and Deputy Chairperson of SAMEA; Nhlanhla Sithole, Assistant Director, National Department of Public Works and SAMEA member; and Annette Ching’andu, Associate Researcher, Khulisa Management Services and SAMEA member.
The Annual Evaluation Conference of the United Kingdom Evaluation Society, held from 24 – 25 November in Bloomsbury, London, was a coalescence of knowledge and practice from some of the most brilliant and seasoned minds in Evaluation in the UK. Under the theme “Evaluating impact in the Current Crisis”, evaluators were simultaneously praised and admonished; appeased and challenged by presentations covering a range of topics from “how to stop worrying and love economics ”, to the “impossible dream ” of determining impact in some settings. The format of the conference included both Panel and plenary sessions around themes such as evaluating social impact, economic impact, environmental impact, health impact and the impact of the arts. A number of presentations and discussion sessions also looked at the interface between evaluation, research, policy and practice in both the developed and developing world, with a particular focus on the role of the public sector in the latter. Echoing the aim of UKES, the purpose of the conference was primarily to advance and improve the theory and practice of evaluation through dialogue.
The three South African delegates fully used the opportunity of engaging with an international evaluation community spanning both the developed and developing world, and a variety of challenging and insightful questions and comments were posed to presenters throughout the conference. The nature of each SAMEA delegate’s work in the field of M&E (and particularly evaluation) in their respective organizations meant that ideas were constantly being tested amongst each other and with other delegates, oft times leading to a discussions and impromptu comparative assessments with the South African context.
The purpose of the conference was to provide a platform for the evaluation community to ask itself some critical questions in the light of the global economic crisis. One – deceptively obvious – question posed by evaluators the world over retained its currency in this conference, namely: how to evaluate impact. Though it could seem slightly incredulous that the evaluation community should have been asking itself this question, the financial crisis, from which the world is still reeling, has created a demand amongst governments and donors alike for proof of the value-add of evaluation in answering and addressing some of the key socio-economic challenges facing the world today.
Though resource constraints have always been a challenge for the voluntary and not-for-profit sectors, many governments and donor agencies had in the recent past been willing to fund M&E work at great cost, creating a rapidly growing industry for both the not-for-profit and private sectors. In a sense, the evaluation community may have been anaesthetised against the looming financial crisis, and the lethargy in the financial world created a context where budgetary issues were overshadowed in evaluation bids. Now, determining just what contributes to creating fundamental changes to the development challenges facing society is critical to demonstrating the value of evaluation in this recessionary economic climate, from which the discipline must protect itself. Moreover, Evaluation plays a critical role in protecting the needs and interests of the most vulnerable who have borne the brunt of the economic crisis most severely by demonstrating the value of the continued funding of development programmes and projects. For reasons more important than the insulation of the profession from the recessionary climate, the evaluation community has to ask itself critical questions around demonstrating the impact of development programmes to bringing about change amongst the poor and marginalized in society, particularly in the developing world.
The Conference provided an opportunity for evaluators from across all disciplines and sectors (government, academia, the voluntary and private sectors) to attempt to provide answers to this question of impact. Case-studies were by far the most popular of the formats for presentations, re-affirming the notion that it is perhaps less easy to generalize the results of evaluations across different contexts and situations. Of particular interest to the SAMEA group was the use of practical tools and methods in evaluating social and economic impact, the development and utilization of evaluation frameworks, and the development and promotion of evaluation in the context of the New Public Management paradigm dominating the public sector landscape in South Africa.
2. Value of the visit
All three bursary recipients have had unique opportunities to disseminate information and lessons learned from their participation in this conference. In particular Ms Morkel, as deputy chairperson of SAMEA and Director: M&E in the Eastern Cape Provincial Government has already engaged a number of key individuals to support the hosting of seminars on important evaluation issues, including the Nelson Mandela Metro University and the Office of the Public Service Commission. Official follow-up discussions with the UKES have already been held in order to develop a partnership between the two associations to develop these kinds of opportunities and explore others. A piece about the three South African delegate’s experience of the conference was also accepted for publication in an edition of SAMEA Updates (the association’s e-newsletter) as well as The Evaluator, the official magazine of the UKES.
The opportunity that the UKES conference provided was not only the short-term attendance of the conference, but the opportunity for networking and of forging relationships towards support for improving the capacity for evaluation in the South Africa through SAMEA. The opportunities for information-sharing and potential for future participation in cross-national research & other activities were also key benefits of this conference.
3. Format and themes of the Conference
The theme of the conference – ‘Evaluating impact in the current crisis’ – was meant to take participants back to basics by posing the question that perpetually taxes evaluators, i.e. how to evaluate impact. Due to the economic crisis that had reached its apex in 2009, evaluators across the world were asking themselves why evaluation did not seem to be providing the answers to the critical questions about development impact that were needed. The pressure placed on evaluators to ‘prove’ development impact and to link successful outcomes to specific development interventions was becoming more pronounced due to increasing funding constraints, which brought about a renewed challenge for evaluators to demonstrate value for money.
The conference was designed around six themes: Evaluating Economic Impact; Evaluating Social Impact; Evaluating Health Impact; Evaluating Cultural Impact; Evaluating Environmental Impact; and the Impact of Evaluation on Policy and Practice.
Theme 1: Evaluating Economic Impact
This theme explored the techniques available for evaluating economic impact, and analysing costs and benefits. One of the presentations discussed and evaluation of the relative effectiveness and cost-effectiveness of public and private prison probation services explored how development decisions can be made using the tools of economic analysis. However, there were many limitations to the research findings that were raised, such as contextual issues as well as internal and external validity.
Theme 2: Evaluating Social Impact
This theme was raised due to evaluators increasingly being tasked to demonstrate evidence of social impact against outcomes such as well-being, cohesion and personal development. It explored the benefits and pitfalls of evaluating social impact on targeted beneficiaries, as well as the ethics of measurement and the delivery of a robust evidence base.
Theme 3: Evaluating health Impact
The issue of measuring health impact drew evaluation examples ranging from a variety of contexts for example measuring health outcomes in skills and employment programmes, to the evaluation of complex health improvement programmes. These elicited discussions around the policy context, methodological challenges as well as ethical issues, and how evaluation results ultimately improve the health and well-being of society.
Theme 4: Evaluating cultural impact
There is an increasing demand for evidence demonstrating the impact of arts and cultural interventions, especially in the context of funding constraints and pressurized budgets. Methodological challenges remained the main issue of concern, as measurement of the outcomes of these interventions is inherently difficult. An example of a presentation made in this theme was an exercise in valuing engagement in culture and sport, using cost-benefit analysis. The study concluded with a map of the types of value associated with increased engagement in culture and sport, the empirical literature available to estimate these values as well as the gaps in this literature. The purpose of the map is to enable users to identify which elements of culture and sport are amenable to valuation using economic techniques, and which need other methods.
Theme 5: Evaluating environmental impact
Climate change, water pollution and water crises, as well as the various threats to sustainable livelihoods on a global scale prompted the highlighting of this strand of the conference. A presentation entitled “a tale of elephants and bees: evaluating very different beasts” captured the essence of the discussions in this thematic area, although it was focused on a specific Waste and Resources Action Programme based being implemented in the UK. It became clear that the issue of the utility and utilisation of evaluation findings is critical in evaluating the achievement of environmental goals, and critical questions were asked about whether the current over-reporting on environmentally efficient behaviour actually indicates success. In this respect, the issue around skewed measurements so as to be seen “doing the right thing” may be defeating the achievement of environmental outcomes.
Theme 6: Impact of Evaluation on Policy and Practice
This thematic area explored the issue of how evaluation impacts on policy and practice, and whether this impact could be measured. Many of the discussions centred around the conflict that often exists between consumers or users of evaluations and evaluators, and building these relationships to maximize the utilisation of evaluation findings in policy and practice. A robust panel discussion session entitled “Evaluation in Whitehall” chaired by Prof Colin Jacobs of the British Council and UKES Board member elicited many differing opinions around what is needed to enhance the implementation of evaluation findings in the public sector, mirroring the many similar questions being asked in South Africa as we embark on our process of creating the space for evaluations to influence decision-making.
A round table discussion was also held to share learning and to provide opportunities for future international networking, and the majority of participants were from developing nations. One of the major challenges that were highlighted is that developing nations face significant challenges around data quality, authenticity of information and the reliability of information from monitoring and evaluation for influencing policy and practice, as well as the institutionalization and sustainability of M&E systems within political and/or administrative systems that had not yet matured.
4. Observations
Columnist, author and economist Will Hutton provided a riveting start to the conference with his presentation on the global melt down in the financial markets to explain the dilemmas faced by evaluators today. Using examples, he described how some members of the financial community - much like evaluators - had raised the alarm of the impending crisis years ago. For their efforts they were ignored, and in some cases were left with no option but to resign. He used these illustrations to challenge evaluators to think about their “early warning” role, to question the notion of being an objective researcher and bystander that produces information, and to emphasise the need for evaluators to voice all findings truthfully, even those which may not be all that popular.
Head of 3ie Howard White, in his presentation on the rise in Impact evaluation in developing countries, gave examples of the growing currency of evaluation in countries such as Mexico, which has legislated that all social programmes be subjected to an evaluation. He also emphasised the need to unpack the causal chain in evaluation, and to understand the context within which it occurs, taking into account appropriate counterfactuals. An interesting point raised in the Q &A session was that current evaluation tends to be “extractive” rather than “instructive”, and that participants (or beneficiaries) are usually left out of the process.
Perhaps not surprisingly, the perennial problem of attribution was debated across a number of parallel sessions, once again dividing the evaluation community (at least for the duration of the sessions) along the fault-line of positivism and post-positivism in demonstrating causality. Largely, though, the resounding message that appeared to gain momentum amongst the delegates is the importance of a mixed-methods approach to evaluation – while still ensuring the maintenance of rigour and robustness in the execution thereof. What is to be avoided at all costs is what was coined a “venture-capital approach to evaluation”.
5. Conclusion
The conference reinforced the role of associations such as SAMEA as key players in the strengthening the knowledge-base and practice of Monitoring and Evaluation, and was extremely valuable for networking and strengthening ties between SAMEA and other evaluation societies and associations. As a result of the dialogue, the potential for future participation in cross-national research & other evaluation activities abound, and the door has been opened for future partnerships between SAMEA, UKES and other Evaluation associations.