Insights
In our previous blog, Lynne Miles explained how the evolution of our review process led us to revisit our first four systematic reviews, and update them to bring them in line with our later reviews.
The new methodology has not translated into big changes in our findings on impact, but for clarity we think it is worth pointing them all out for anyone who is making extensive use of the original papers. Here is a summary of the impact that the changes have had on the findings.
Employment training – broadly in line with initial findings
The new version of the report shows that training has a positive impact on participants’ employment in nearly half the evaluations, and on participants’ wages in just over half the evaluations (though in several cases it is noted that wages do not increase in the immediate term, but only after a number of years). This is broadly in line with our headline finding in the initial report, despite some changes in the classification of individual evaluations (for reasons discussed in Lynne’s blog).
No changes were made to the headline findings for individual programme design elements, though we did correct a small number of mis-categorisations. In one case, by splitting out the outcomes for employment and wages we were better able to justify the headline finding that in-firm / on-the-job training programmes outperform classroom-based training programmes.
Business advice: Strengthened positive impact on sales and turnover
The new version of the report shows some differences in headline findings for certain outcomes, due to changes in the way the evaluations are classified (as discussed in Lynne’s blog). Most notably, whereas before, the impact of business advice was found to be more positive for productivity and output than for employment, the results have shifted to indicate that results are in fact better for firm sales and turnover than for either productivity or employment. The primary reason for this is a shift in classification of some of the results for GVA / productivity from positive to mixed and zero. The changes to the way in which the results are classified had less of an effect upon results for employment, sales and turnover (with the latter two now having the largest proportion of positive findings in the new version).
The change in how outcomes are categorised has also resulted in a slight fall in the number of studies which show a positive impact on at least one business outcome, decreasing from 17 out of 23 evaluations to 14 out of 23.
Sport and culture: Weakened evidence of positive impact on wages and income
The new version of the report shows a difference in headline findings for one outcome, due to changes in the way the evaluations are classified (as discussed in Lynne’s blog). In the original report it said that positive effects on wages and incomes were slightly more likely than positive effects on employment. With the alternative classification the evidence no longer supports this. The primary reason for this is a shift in classification of some of the results for wages / income from positive to mixed and zero. To reflect this, the updated report simply states that effects on wages and incomes tend to be limited. The changes to the way in which the results are reported have had less impact upon other outcomes.
Access to finance: Weakened evidence of positive impact on credit availability and investment
For some outcomes there were shifts in the balance of findings from positive / zero to mixed. This resulted in notable changes for credit availability (three positive results now classified as mixed) and investment (one positive result and one zero result now classified as mixed). There are also single papers which have been reclassified as mixed for employment, wages & income and sales & turnover.
The new papers have replaced the originals on our website. Do get in touch if this raises any problems or questions.
