In our discussions around the country about how to better understand the impact of economic growth investment, we always recommend Randomised Controlled Trials (where feasible) as the very best approach to evaluation. One of the most common concerns raised is the ethical considerations of running a control group, when this means offering participation to only part of a cohort that needs support.
Jemma Mouland of Family Mosaic sums up this dilemma perfectly in her recent blog for our website:
"The first and least surprising hurdle was the control group. As a housing and social care provider we come from an environment where we are used to providing support to anyone who needs it. The idea of a control group was a completely foreign concept and went against many of our values. It took time to persuade staff that our methodology was essential to making valid statements around the impact of our services and making a difference in the long-term."
Although a commitment to fairness would dictate that everyone should be able receive the same ‘treatment’ or programme support, a universal approach will not tell us whether a programme is working the way we hope it will.
Here is why we tell people that the ethical concerns – although understandable – should be overcome:
- Until we know that these policies work, we should not assume that the group being treated is better off. There are many examples of policies which were proven by RCTs not to have been as beneficial as previously assumed. We should not be spending public money on large rollouts of programmes that have not been tested and proven to be better than existing programmes, or better than doing nothing at all.
- The medical community have been running trials with untreated controls for a generation and they are accepted as an integral part of the scientific method, and a required part of approval for new treatments. The fact that they are accepted in a situation where life and death are actually at stake suggests that the benefits outweigh the concerns – and we in the economic growth community should take comfort from that.
- There are often natural ways to construct a control group, for example when a new programme is oversubscribed, has limited geographic boundaries, or staggered start dates.
Where there is considerable opposition (often, and understandably, from politicians) to finding an untreated control group, there can be useful lessons learned from experimenting with different approaches between two randomly allocated groups to see how results differ between them.
No doubt this question will continue to arise – on the surface the use of a control group goes against the grain of so much that government is about. But the law of unintended consequences tells us that we cannot assume that programmes are going to work the way we hope that they will. We have to test them first.