Skip to content

How to evaluate business advice: Export promotion initiative (RCT, SMS level 5)

arrow down
denis-volkov-IvsHiuYhqus-unsplash

What was the programme and what did it aim to do?

This study evaluates the impact of an information intervention by UK Trade and Investment (the UK’s Export Promotion Agency, EPA). The pilot provided firms with information in the form of a cover letter and a brochure which consisted of three sections: 1) a list of benefits from exporting, as reported by other firms; 2) details on potential barriers and the most relevant UK Trade & Investment (UKTI) service; and 3) a description of the UKTI services available to help firms with exporting.

The main aim of the pilot was to encourage firms to begin exporting or increase their exports. The thinking was that some firms were not exporting due the lack of accurate information on the costs and benefits of exporting, and the services available to them. The trial was testing whether low-cost information provision could address these problems. The major focus of the evaluation was the effect of the information on perceptions, as well as actual exports.

What’s the evaluation challenge?

Evaluating the impact of business support policies such as export promotion efforts is difficult because firms who choose to participate in these programmes, or look for support from EPAs, tend to be different to those who do not seek assistance, in ways that are hard to observe or measure. As a result of this selection, if we compare differences in outcomes for firms that seek assistance to firms who do not, the differences in outcomes would not necessarily reflect the impact of the programme. Instead, they may simply reflect characteristics of firms who selected to receive this type of support (e.g. motivation, quality of management, size of network). Therefore, the evaluation challenge is to isolate the true impact of the programme from the unobserved characteristics of those firms requesting support.

What did the evaluation do?

The evaluation deals with this selection on unobservables by implementing the pilot as a randomised controlled trial (RCT). A first stage selected a representative sample of 6,015 UK firms in the manufacturing sector, with between 2 and 250 employees (thus excluding self-employed and large firms not eligible for this type of support). From these firms, 50% were randomly selected as a control group, while the rest received information on the costs, benefits and services to help them with exporting. All the observable firm-level characteristics that could play a role in explaining exporting are balanced between the control and treatment groups. Thus, the mean difference between the treated and the non-treated firms cannot be attributed to firm characteristics, but rather should be attributed to the intervention.

The study implemented a baseline survey to capture perceptions on cost and benefits of exporting and conducted a follow-up survey after intervention. The evaluation compares the difference in perceptions between treated firms before and after the intervention, to the difference in perception for those who did not receive the information during the same period. This method is known as a difference in difference, implemented here with additional randomisation across the groups to help control for anything else that might have driven changes in perceptions.

In addition, the study matched the RCT sample to UK customs data provided by HMRC to get information on actual exports of firms. It uses this data to look at the cumulative effect of export value after the intervention comparing control and treated firms, while taking into account the initial level of exports and fixed effects (e.g. quartile of assets by industry).

How good was the evaluation?

According to our scoring guide, randomised control trials can achieve the maximum score of 5 on the Maryland Scientific Methods Scale (SMS) because randomisation balances out the observable (age, size, assets, industry) and unobservable (e.g. effort and quality of management) characteristics between treated and non-treated firms. In this study, the randomisation worked satisfactorily: treated and untreated firms were not statistically different on 33 out of 35 observable characteristics captured in the survey, and on 5 out of 5 measures from the customs data (including export status, total firm exports, the number of destinations served and the number of products exported).

Another important challenge in a RCT is attrition, as differential attrition between treatment and control groups could signal that they differ in unobservables. Fortunately, the study finds no systematic attrition in terms of observable characteristics.

Overall, we score the study at the maximum of 5 on the SMS.

What did the evaluation find?

The evaluation finds that the intervention had a different effect for firms already exporting versus firms that were not exporting. Exporting firms experienced a positive and significant cumulative effect on export value up to about 9 months after the intervention; mainly in their existing products and destinations. After a year, the programme had increased cumulative exports for initial exporters by around £165,000. In contrast, there was no effect for non-exporting firms. These export results are consistent with the effects on perceptions, which suggests that the information intervention reinforced existing views. The perceived benefits and costs deteriorated for the group of treated non-exporting firms, while the intervention made exporter firms more optimistic.

What can we learn from this?

This study provides robust evidence of an asymmetric response of firms to an information intervention aimed at modifying beliefs about the costs and benefits from exporting, and promoting actual exports. The results show that the programme is only beneficial to firms that are already exporting, who increased the value of their exports.. However, the new information might lead non-exporters to realize the potential profit from exporting was smaller than previously believed. Thus, targeting exporting firms with this type of intervention might be optimal, while non-exporters may require other type of policies to change their perceptions and encourage them to export. For existing exporters, this is a potentially cost-effective way of promoting exports, and it would be easy to scale up the programme.

Reference

Breinlich, H., Donaldson, D., Nolen, P. J., & Wright, G. C. (2017). Information, perceptions and exporting-evidence from a randomized controlled trial. Working paper.