Improving evaluation and appraisal

Evidence gaps

Improving evaluation and appraisal

In many instances local economic impact is an important part of the case for transport investment. Such investment also forms a central component of many governmental policy initiatives aimed at increasing local economic development (e.g. the UK government’s Local Growth Deal process). It is therefore vital that progress is made in filling the evidence gaps and in improving our understanding of the effect of transport improvement on local growth.

Here you can find some preliminary recommendations building on recent work with the Department for Transport, as well as the discussions of a LEP working group convened by the What Works Centre for Local Economic Growth. Our recommendations focus on the need for more, and better, ex-post impact evaluation and the need to embed such evaluation in to the appraisal process.

Considerable resources are already devoted to the ex-ante appraisal of transport schemes as part of the decision making process. Cost-benefit analysis plays a central role in such appraisals. The increased interest in the effect of transport investment on the local economy has also been accompanied by growing criticism of the appraisal approaches used to help facilitate scheme prioritisation. In particular, there is growing criticism that the current approach to cost-benefit analysis does not capture all of the benefits that may be associated with transport investment.

If the main aim of new transport infrastructure is faster journeys, then benefits to the economy materialise because time saved can be used on productive or otherwise valuable activities (either in business or leisure). This is why the most fundamental input into transport infrastructure cost–benefits analysis has traditionally been the so called ‘value of travel time savings’. This is travel time saved, converted into monetary units. These monetised time savings are a crucial measure of the economic benefit from transport investment (and can be supplemented by monetised estimates of the benefits of reductions in other costs like accidents and unreliability).

But over the past 15 years there has been greater interest in the potential for transport to generate ‘wider economic benefits’ that go beyond these travel times savings (in addition to a range of other wider social and environmental benefits). In the context of UK appraisal a particular focus, in terms of wider economic benefits, has been on those that come from effectively bringing people and businesses closer together to form agglomerations of economic activity. The logic follows from the observation that cities are more productive than rural places and big cities are more productive than smaller cities. So linking places together may help generate productivity improvements.

Despite these improvements to appraisal practice, there continue to be concerns that appraisal misses important benefits of transport investment – particularly in terms of the impact on local economic growth. Some of these issues were recently considered in an independent report for the Department for Transport (Laird, Overman and Venables, 2015). The report concludes that, in some cases, traditional cost-benefit analysis may indeed miss important benefits that should be included in the analysis (although the Department’s WebTAG guidance includes almost all of them).

The report also argues that there may be instances in which local decision makers are interested in the local economic effects of transport – e.g. on employment and investment – even when these should not be included in a cost-benefit analysis which seeks to evaluate the overall (i.e. national) gains from a new project. A concrete example would occur when employment growth near to new transport investment is purely driven by displacement from elsewhere in the economy. A traditional cost-benefit analysis – which tries to assess the overall gains to society – would ignore such displacement. But these effects may be of legitimate interest to local decision makers.

DfT is planning to refresh WebTAG (the set of procedures which outline how appraisals should be conducted) to respond to these observations. While such an exercise will be welcomed by many, the findings in our review also highlight the importance of complementing any further work on the ex-ante appraisal framework (i.e. analysis to predict what might happen) with additional work to strengthen the ex-post evaluation of transport investment (i.e. analysis of what actually happened). Not least because, as this report makes clear, there is limited evidence that the employment (and other effects) that would underpin any changes to the guidance actually occur in practice.

What form should such ex-post evaluation take? As discussed above, for capital expenditure, where investments are durable, it is hard to imagine situations in which true randomisation of project placement would be either feasible or desirable. This means that we need to rely on alternative evaluation approaches to try to identify the causal impact of transport investment. On the basis of our review work, plus our wider work on the issue of transport evaluation we think that work to develop a new approach is urgently needed. Any such approach needs to be both feasible and proportional. It also needs to produce evidence that is helpful in improving future decision making. Unfortunately, many existing studies appear to have cost much but with arguably little benefit in improving decision making. There are a number of possible avenues that can be explored and issues that will need to be considered. For example:

  1. At present the Highways Agency undertakes Post Opening Project Evaluation (POPE) of a large number of schemes. This takes two forms – one, lighter touch, for smaller projects (Local Network Management Schemes); and a more extensive evaluation for larger projects (Major Schemes). POPE aims to determine how schemes have performed in their opening year and, for major projects, five years after opening. Findings are compared to ex-ante appraisal to assess accuracy and an annual meta-study pulls together findings from all POPE undertaken that year. POPE uses before and after analysis of scheme data – an approach which would score level 2 on the SMS scale. More recently, the DfT has issued a Monitoring and Evaluation Framework for Local Authority Major Schemes (LAMS) that provides guidance that outlines a POPE style approach for those schemes. This guidance outlines three approaches: standard and enhanced monitoring which parallel the smaller projects approach in POPE; and fuller evaluation which is closer to the POPE guidance for Major Schemes. As with POPE, there is a strong emphasis on before and after comparisons.
  2. It would be helpful to consider how the use of appropriate control groups could refine these processes. Interestingly, the most recent POPE guidance for major schemes has recognised the importance of controlling for the background reduction in the number of collisions when assessing benefits from accident reduction. LAMS also places more emphasis on the use of control groups – at least for the fuller evaluations (see, for example, the discussion of control groups in the assessment of changes in travel behavior). It would be useful to undertake further work to identify appropriate control groups and to encourage their use for benefits which might be most affected by other ‘background’ changes. Control groups could be constructed in a number of ways: For example a) for similar parts of the network that have not been subject to improvement b) from schemes that are likely to be funded in the future but have not yet been funded; c) from schemes that have similar benefit-cost ratios but were declined funding; d) for areas close to funded schemes that are not directly affected by the scheme. More simply, appropriate area wide averages (which would ideally exclude new schemes) could be used to provide a very basic control group. Similar approaches could be used to identify suitable control groups when using individual level data (e.g. on travel behavior). There will, of course, be pros and cons to all of these approaches and further work would be needed to consider the alternatives (and whether the benefits in terms of improved POPE and LAMS analysis, outweigh the additional costs).
  3. Although POPE considers performance of each scheme against the Government’s four WebTAG objectives (economy, environment, society and public accounts) the analysis of economic impacts tends to focus on direct benefits – particularly in the form of reduced accidents and improved journey times. This is unsurprising given the current focus of POPE on performance. Interestingly, while the lighter touch approaches in LAMS give less consideration to some WebTAG objectives they place more emphasis on economic benefits – particularly in terms of effects on employment and rental values. POPE for major schemes does allow for the assessment of wider economic benefits with the consideration given to these benefits varying according to the level of POPE that is undertaken. At its most extensive, this will involve a survey of local business at one year to ‘identify emerging concerns or positive outcomes associated with the scheme’ and at five years a ‘focused survey of businesses to identify wider economic impacts’. Given the increased interest in wider economic effects, we think that this process could be improved to better align the POPE and LAMS processes to ensure that both carefully assess these economic effects. As is made clear in LAMS, not all schemes would warrant such an analysis, but this should be considered when employment, or other local economy effects, are an important component of the strategic or economic case for major schemes. Results from this analysis are unlikely to be useful in isolation. Both LAMS and POPE highlight the importance of comparing outcomes to key appraisal assumptions. But we would also highlight the importance of broader comparisons to both the strategic and economic cases that form part of the appraisal process.
  4. Once again, it will be important to consider how the use of suitable control groups could play a part in the analysis of these economic effects. As LAMS recognises, there are strong arguments in favour of developing such an approach – at least for larger schemes. Some of the options for constructing such control groups were discussed under point (2). It would also be useful to consider whether a light touch approach could be developed for smaller schemes.
  5. There needs to be a much closer link between the ex-ante appraisal and ex-post evaluation of schemes. Our review of the literature discovered a large number of ex-post evaluations that appear to live in a vacuum, with no attempt made to link the findings from these reports back to scheme appraisals. Higher quality impact evaluations – i.e. those that seek to identify the causal impact of investments using changes in outcomes compared to a control group (i.e. are scored SMS 3 and above and included in our review) – are still helpful even in the absence of such comparisons. These are the studies that we have used in our review. This is, unfortunately, not so true for less robust evaluations (e.g. those involving simple before and after comparisons). Embedding evaluation in to the scheme prioritisation process is an important step in ensuring that money spent on ex-post evaluations is cost-effective in improving prioritisation for future spending. One of the advantages of incorporating the evaluation of wider economic impacts within an improved POPE methodology is that these comparisons are already part of the POPE ‘meta-analysis’ process. It will be important to develop a similar ‘meta-analysis’ for LAMS that parallels the POPE process. These comparisons across evaluations should allow findings on scheme effects and the comparison to appraisal assumptions to be used to improve scheme prioritisation (for example, through the use of optimism bias to adjust predicted employment effects). Given the interest in the economic impacts of investment in other transport modes we should consider how and when a similar approach could be extended to such schemes not covered by POPE or LAMS.
  6. In line with the recommendations of the DfT TIEP report, there should be far greater attention paid to the critical analysis of both the economic and strategic cases for support. This critical analysis should occur both ex-ante (on the basis of available evidence – including that covered in our review) and ex-post (on the basis of appropriate evaluation – including analysis developed according to the recommendations above).
  7. Ex-post evaluation needs to include due consideration of the extent to which any employment effects are likely to result from displacement (the shifting of jobs from one place to another). Addressing concerns over displacement will need to be a key question in understanding the net impact of investments. There are similar concerns over spillover effects whereby employment growth resulting from transport improvements drives growth in areas not directly affected by the scheme. However, for both these questions carefully identifying any direct employment impact is a first step in understanding these other factors. Concerns over displacement and spillovers should not prevent progress in asking the simpler question as to whether any changes in employment occur directly as a result of the scheme. Work is needed to tackle all three questions – what is the employment effect; are any local employment changes additional; do these spillover to wider areas? A similar point holds with regard to variations in effects across schemes. Work to identify the average effect should be a first step in understanding how variation in effects depends on context. Again, concerns over heterogeneity of effects should not prevent progress on the simpler question of identifying average effects.
  8. The current LAMS guidance, including a comparison to appraisal assumptions, would appear to provide an appropriate framework for undertaking and improving evaluation and scheme prioritisation for individual LAs/LEPs. However, there is a role for DfT in helping develop the guidelines for how this analysis could be conducted and improved along the lines of points (2) to (7). This will ensure, as with POPE, that results for specific LAs/LEPs are transferable across areas. The devolution agenda raises questions about the extent to which such an approach could (or should) be mandatory. Regardless of the outcome of that debate, many LAs/LEPs would still welcome guidance on how best to proceed – especially given local constraints on analytical capacity.
  9. Consideration needs to be given as to how to ensure LAs/LEPs have the incentive (and the resources) to collect data in control/comparison areas. It is possible that central government departments could provide appropriate area data (and the use of such secondary data would substantially reduce the cost implications of undertaking evaluations). Where data is collected at the local level it will be important to ensure that such data are available to researchers for use in aggregated/multi-intervention analysis.
  10. Given the complexity of many of the issues raised above, and the need for comparison across areas, it is likely that DfT will need to play a coordinating role in addressing many of these evaluation challenges. There is also a role for DfT in undertaking multi-intervention ex-post analysis using the kind of approaches used by the higher quality studies considered as part of this report.

Further work would be needed to develop the issues discussed here and to consider appropriate solutions. It is crucial that further work recognises the importance of embedding evaluation in to the scheme prioritisation process (DfT is currently undertaking work on this issue). This means better aligning appraisal and evaluation, particularly if the objective is to improve scheme prioritisation. Without closer integration, there is a danger that we undertake refinements to the appraisal process – e.g. to include employment and investment effects – without knowing the likely magnitude of effects, whether they are additional, etc. Similarly, while ex-post evaluation can serve some role in terms of monitoring and accountability, its main aim should be in improving future decision making. This means thinking about ways in which evaluation can feedback in to the scheme prioritisation process – both in terms of developing ex-ante appraisal, but also in providing a means of scrutinising strategic cases for future investment (again, an area in which DfT is currently undertaking work).

Related Content

Register for WWG Updates