Previous posts in this series have looked at how good evaluation should be designed into local economic growth policies from the start and how, even with a small amount of planning, this needn’t be technically difficult or expensive. This post takes a look at a part of evaluation that is all too often overlooked: the need to bring the people or organisations you expect to act on the results in along with the evaluation process. Many good evaluations have failed to have the impact they rightly deserve because they missed out this crucial step. Likewise, many poor evaluations have had disproportionate impact because they got this right. There are two key questions that this blog explores: who needs to be “on board” and when.
The answer to the second question is easier than the first: as soon as possible, ideally before the intervention has begun or the evaluation design finalised. The reason to bring people in early is that by far the easiest way to ensure buy-in to the evaluation process, and ensure the results are acted on, is by developing a sense of ownership from the start. A few days delay in finalising the evaluation approach can save months of wrangling over technical details when the results of an evaluation are approaching publication. It can also make the difference between an evaluation being acted on and resulting in real policy change, and it being left sat on the proverbial (digital) dusty shelf.
When engaging with policy-makers early in the life of an evaluation, it is important to remember that most local economic growth policy making is short-term in the UK. Policy makers are unlikely to be interested in the design of an evaluation that isn’t going to report for perhaps two years or more as it doesn’t meet an immediate need. However, once publication is approaching, and the evaluation has the potential to disrupt an agreed policy approach, interest will rise exponentially. It’s best to pre-empt this and take the time to engage widely from the start, despite the indifference that may confront an enthusiastic researcher.
Deciding who to get on board is more complex. In local economic growth policy, the list of actors with a potential stake in an intervention is very broad. Rarely can one organisation act in isolation from its partners, as the interdependencies between different parts of the system are too great. The same is true of interventions and their evaluation. Engagement with four key groups of stakeholders is likely to be necessary:
- The funding organisation: When an intervention is externally funded it is likely that the funding organisation will require an evaluation and will in all likelihood fund it, so for purely pragmatic reasons it is helpful to have them on board with the approach. It is also the case that the funding organisation will usually be one of the key actors that the evaluation will be looking to influence, so their buy-in is essential.
- The delivery organisation: There’s no question that good evaluation requires access to high quality data and in most cases the delivery organisation will collect this information. Getting buy-in from those responsible for collecting evaluation data can make the difference between data input being seen as an unimportant administrative burden and it being viewed as an essential part of project delivery and service improvement. Buy-in from the frontline can also provide invaluable soft intelligence to give more rounded evaluation findings. A business advice intervention may fail because of a fundamental policy design flaw or because the business advisor spent all their time on Facebook. Spending the time building trust with those delivering a service can identify what the real lessons learnt are.
- Partner organisations: In local economic growth it is highly likely that any evaluation is going to identify that not all the benefits and not all the costs of an intervention flowed to the organisation that funded it. More sustainable delivery models can be developed if key partners are included in the evaluation process so that they can get an insiders understanding of the effect of an intervention on their organisations finances.
- Key decision-makers: In local economic growth, local authorities and local enterprise partnerships are likely to be important actors in deciding whether to scale up or scale down an intervention based on the evaluation findings. Early engagement can help ensure that they anticipate the evaluation findings and factor the timings of the results into their policy development cycles.
Finally, take some time to think about the person you will likely probably never meet but that is desperate to be on board with you: the researcher or policy-maker searching for evidence of what works. By making your evaluation findings available freely online you can have a policy impact that goes way beyond the immediate local context.