Evidence review|Evidence briefing
What was the programme and what did it aim to do?
This study evaluates two programmes run by the UK Trade and Investment agency (UKTI). Aftercare provides support for inward investors already in the UK and aims to attract more investment by these foreign firms by helping them integrate within the local economy. The Passport to Export service supports domestic firms to increase their export activity and offers a range of services such as training, access to foreign contracts, and export development services.
What’s the evaluation challenge?
The fact that UKTI chooses which of the applying firms participate makes it harder to assess the programme’s impact. There are two sources of selection bias: firstly, only firms that expect to benefit will likely apply for the program. Secondly, given a limited budget UKTI may be tempted to ‘pick winners’ by selecting firms that look likely to succeed. Overall, this means that participating firms will likely be different in performance to other firms – and this may bias the estimated effects of the programme. For example, if participants were generally larger firms (observable) or had more ambitious CEOs (unobservable) then differences in performance compared with non-participants could simply be an effect of these factors.
What did the evaluation do?
The authors use a difference-in-difference matching approach to estimate the impact of the policy. A control group was constructed by carefully matching program participants with other firms on the basis of variables such as size (employment), productivity (value added per worker), assets and exporting intensity. The difference-in-difference estimate is the before-and-after change in outcome variable for the treatment group (i.e. those that receive advice) relative to this control group.
To implement this approach, firm level data was taken from the FAME database from 1994 to 2003 and linked with UKTI management data. There were two other export promotion programmes that were dropped from the analysis since the linking between datasets for participants of these programmes was considered unsuccessful by the researchers. Their final treatment groups include 299 firms who received Passport support at some point over this timeframe and 509 firms who used Aftercare.
How good was the evaluation?
According to our scoring guide, matching combined with difference-in-differences receives a maximum of 3 (out of 5) on the Maryland Scientific Methods Scale (Maryland SMS). This is because it does well to control for observable differences (e.g. sales) between supported and non-supported firms, but is unable to control for unobservable differences (e.g. motivation). Since this paper uses a wide range of variables in its matching and since the difference-in-difference is based on a clear treatment date we score this study 3 on the SMS.
What did the evaluation find?
The evaluation examines the effect of Aftercare and Passport on a number of outcome variables, covering both core objectives (exports for Passport and investment for Aftercare) and other outcomes such as employment, productivity and profits. Participation in Aftercare was found to increase the level of investment of foreign manufacturing firms based in the UK. In contrast, for foreign service-sector firms there was no effect on investment. Participation in Passport was found to have successfully increased the level of export activity for service-sector firms (but not for manufacturing firms). Passport was also found to increase total factor productivity. Neither programme had any effect on firm employment or profits.
What can we learn from this?
On the face of it, these results imply that the UKTI programmes have been successful in reaching some of their goals, but not others. Effects also appear to vary across types of firms and across programmes. It would be helpful to better understand what is driving these differences, for example through random control trials designed to better understand the effects of these different programmes. Better evaluation methods would also address concerns that the results may be driven by unobservable characteristics of firms that are offered support. For example, firms on the Passport programme are likely to have been actively considering increasing their export activity and as a result may have shown increases compared with the control group even if they had not been selected. Improved evaluation that better controlled for unobservables (through randomisation, instruments or quasi-experiments) would help pin down programme impacts.
Reference
Girma, S., Görg, H., and Pisu, M. (2005). Quantitative analysis and linked micro-data study of UKTI services – Final report. London, UKTI. [Study 169 from our Business Advice review]
