How to evaluate business advice - Subsidised consulting for SMEs in Mexico (RCT)

What was the programme and what did it aim to do?

This study tested the impact of subsidised consulting services on outcomes for small and medium-sized enterprises (SMEs) in Puebla, Mexico. The programme ran from March 2008 to February 2009 and was implemented by the State Government of Puebla. It was aimed at increasing the size of SMEs, thereby creating jobs and economic growth in the region. Participating businesses were matched with an appropriate consulting firm that was asked to diagnose problems that prevented growth, suggest solutions and assist with implementation. The State of Puebla then covered 90% of the cost of services for micro enterprises (defined here as less than 10 employees), 80% for small enterprises (11-30 employees) and 70% for medium-sized enterprises (31-100 employees).

What’s the evaluation challenge?

Evaluating the effect of business advice programmes is difficult because such programmes are used by specific types of firms. Typically, only certain types of firm will apply for support or be selected for support by the programme administrator. As a result of this selection, if we compare differences in outcomes for firms who received advice to those who did not, these differences may not reflect the impact of the programme. Instead, they may simply reflect differences in the types of firms who went on the programme.

What did the evaluation do?

The study addressed the selection problem by implementing a randomised controlled trial. From the 432 SMEs who applied for the programme (following state-wide advertisements), 150 were randomly selected into the treatment group. The remaining 282 firms made up the control group and did not receive any subsidised services. The fact that treatment was randomised makes it much easier to assess the programme’s impact. Specifically, randomising receipt allowed the researchers to ensure the treatment and control groups are similar on both observable characteristics (e.g. age of firm) and unobservable characteristics (e.g. the CEO’s leadership qualities) that might affect outcomes.

How good was the evaluation?

According to our scoring guide, an RCT receives a maximum of 5 (out of 5) on the Maryland Scientific Methods Scale (Maryland SMS). The results of the baseline survey demonstrated that the treatment and control groups were largely similar on observable characteristics, suggesting that randomisation was successful. The rate of response for the follow up survey was around 88%, with no differences across treatment and control groups, nor evidence of compositional shifts due to firms dropping out (‘attrition’). Since the treatment was successfully randomised and attrition was not a big problem, we score this study a 5 on the Maryland Scale.

What did the evaluation find?

This field experiment found huge returns to subsidised consulting. Monthly sales increased by around 80% and profits increased by 120% in the treatment group compared with the control group. The average increase in profits is estimated to lie between $7,600 and $11,000 per month, compared with a cost of $988 per month for the consulting services. In our main review we find that business advice programmes have a mixed record on improving employment, and in this case the researchers do not find any increase in the number of workers employed by treated firms – although perhaps employment may take longer than two years to adjust.

What can we learn from this?

The cost-effectiveness of the programme raises the question of why firms didn’t invest in these services themselves in the first place? The researchers suggest the most likely source of market failure in this case is funding constraints. Another possible explanation is that they simply never considered taking up consultancy services or did not know such services were available. In terms of relevance to UK policy, these results highlight the potential for worthwhile intervention in the area of business support. However, such a policy would be successful in the UK only if it were addressing a similar market failure such as a lack of funding/information for SMEs. Since the Mexican economy is significantly different to the UK economy it may be that the policy would not be so successful. The only way to know for sure if it would be effective in the UK is to replicate the programme here as a randomised trial.

Reference

Bruhn, M., Karlan, D.S., and Schoar, A. (2012) The Impact of Consulting Services on Small and Medium Enterprises: Evidence from a Randomized Trial in Mexico. Yale University, Department of Economics. [Study 162 from our review on Business Advice]

Read more case studies on business advice: Supporting unemployed people to start businesses (RCT) | UKTI Programmes (statistical approach)Regional Business Development Programme in Sweden |see all

Downloads

Related Content

Register for WWG Updates