One of the hardest things about doing our systematic reviews was identifying all of the evaluations that apply in a specific area. Our team has had to plough through 120,000 documents trying to identify high quality evaluations that can inform our evidence reviews. Even then, as we’ve discussed many times before, the number of high quality evaluations is only a small percentage of that total. Learning from evaluation would be so much easier if we simply knew what evaluations were going on.
In medicine, there are major efforts underway to try to fix this. We’d draw attention, in particular, to the campaign for all clinical trials to be registered. There are lots of reasons why this is an important campaign, which you can read more about here.
We aren’t medical doctors, but we share the concern raised by this campaign about burying the results of programmes that do not run according to plan. One of the principle barriers to proper evaluation of economic growth policy is fear of potential political fallout if a programme does not deliver as intended.
It is a problem when a programme does not deliver the outcomes which were hoped for, but it is a complete loss if we are not allowed to ask what went wrong and why. Too often a lack of evaluation means that we do not even know whether a programme worked, and whether it should be rolled out further or scrapped.
Fortunately a step is being taken in the right direction in a policy area closer to our field. The Innovation Growth Lab at Nesta is running an online database of RCTs on innovation, entrepreneurship and business growth.
Putting all of the best evidence together in one place makes it easier for policy makers – who would rather concentrate on their day job of implementation – to quickly get a handle on what the analysis says works best. We would love to see this sort of database for all of the policies aimed at local economic growth. Although it would put us out of a job.
In the meantime we hope that All Trials will help remind professionals in all areas about the virtues of learning from failure. Many of us have signed the petition, please consider signing it too.