Statistical approach (SMS level 3)
What was the programme and what did it aim to do?
This study evaluates the impact of place-based “Promise” programmes in the United States. Promise programmes guarantee financial aid to attend university or college for eligible students within a particular school district. The initial Kalamazoo Promise paid all fees for four years for students to attend any of Michigan’s state colleges or universities, and was available to all high school graduates from public high schools in Kalamazoo, Michigan. Subsequent Promise programmes varied significantly in terms of programme design. For example, the maximum scholarship amount varied from $600 to $10,000 and the duration from one to five years. There was also variation in eligibility, with some being universal like the original Kalamazoo programme and others introducing requirements such as a minimum GPA score. Promise programmes are intended as a place-making policy that boosts local economic development by attracting and retaining parents with high human capital (who would like their children to benefit from the financial aid).
What’s the evaluation challenge?
Evaluating the economic effects of area-based interventions such as Promise is difficult because areas that receive support are different to those that do not. In this case, Promise programmes targeted the most distressed communities in the United States. As a result of this selection, if we compare differences in school enrolment or house prices for Promise areas to other areas, these differences may not reflect the impact of the programme. Instead, they may simply reflect differences in the type of areas that receive Promise programmes.
What did the evaluation do?
In order to address this issue, the study compares the before-and-after change in school enrolment and house prices for Promise areas (‘treated’ areas) with areas that were similar on pre-intervention observable characteristics (e.g. demographics) but that did not receive Promise support (‘control’ areas). This method is called the difference-in-differences approach..
How good was the evaluation?
According to our scoring guide , the difference-in-differences method receives a maximum of 3 (out of 5) on the Maryland Scientific Methods Scale. This is because although it does well to control for observable differences (e.g. neighbourhood demographics) between supported and non-supported areas it is unable to control for unobserved differences (e.g. how accessible jobs are from the area) that change with time. The study ensures the control group is as similar as possible to the Promise treatment group by considering only nearby areas and controlling for remaining observable differences in the regression analysis. Furthermore, there is a clearly defined treatment date. Therefore we score the study a 3 out of 5 on the SMS.