Insights
We’ve updated our ‘scoring guide’ that explains how we assess and score different evaluation techniques. We initially developed the scoring guide as a tool to help us sifting the evidence for our reviews where we use studies that scored a ‘3’ or higher on the Maryland Scientific Method Scale (SMS). Lately, we’ve also been using the guide to help score studies for our toolkits where we used SMS ‘2’s and above. We think the guide should also be useful for policymakers in a number of different ways:
1. Judging existing evidence
You come across an evaluation that is relevant for a policy you are interested in. You want to know how much weight to place on these findings relative to other impact evaluations. If the guide scores the approach as an SMS4 or SMS5 you may want to place more weight on it than evaluations scoring SMS1 or SMS2. Of course, many other factors such as context may matter but the score gives you one indication of how much weight to place on the findings.
2. Selecting your own method
If you are doing an evaluation yourself, you can use the scoring guide to help you develop your approach to evaluation (or assess methods proposed by outside evaluators). Ideally, you want to adopt the highest scoring method that can feasibly be implemented. If you are already a little further down the process, you could use the scoring guide to help choose the most robust method from a set of options. Finally, if you are dead set on a particular method, we suggest you use the guide to get an idea of how much weight you will want to put on your findings (see point 1.)
3. Carrying out your evaluation
Let’s say you’ve selected a method that you’ve not used much previously. It might be that you’ve decided to go for a difference-in-difference this time (good choice! It’s an SMS3) rather than the usual additionality survey (‘not SMS scoreable’). If you’re unsure how to carry out a diff-in-diff we suggest looking at the examples in our scoring guide as a first point of call. As with our series of how to evaluate case studies, these examples cover the essential details of how an evaluation is carried out. Also, as we suggest here, you should look at the studies themselves and copy how they do the evaluation.
The scoring guide is one of a number of evaluation resources available from the Centre. Try taking a look at our guide on how to evaluate or the evaluation case studies (within the resources). We’re always happy to provide further advice, so please get in touch if you have questions.
