How to evaluate broadband – US Department of Agriculture Broadband Loan

What was the programme and what did it aim to do?

This study evaluates the impact of the USDA Broadband Loan programme in the United States. The programme was designed to increase broadband access in rural areas underserved by private telecommunications operators. In 2002 and 2003, the government dispersed around $180 million dollars in loans for small telecommunications operators to expand broadband access. Interest rates were subsidised and companies had to match at least 15 per cent of the value of the loan. Despite the fact that only rural areas with fewer than 20,000 people were eligible for the loans, the overly broad definition of “rural” implied that in practice, many of the loans were dispersed to suburban communities located around large urban centres. Furthermore, the programme’s pilot and subsequent expansion differed in the stringency of loan conditions. Given that one quarter of the pilot loans defaulted, subsequent loans involved higher equity and security requirements.

What’s the evaluation challenge?

Evaluating loan programmes such as these is difficult because the areas that received the loans are different to those that did not. In this case, the programme targeted areas that did not already have broadband infrastructure. These areas were also sparsely populated and rural. Furthermore, since private telecoms operators had some discretion in choosing which areas to serve, they likely prioritised areas that were economically appealing (e.g. more educated or wealthier). As a result of this selection, if we compare differences in outcomes for treated areas to outcomes for other areas, these differences may not reflect the impact of the programme. Instead, they may simply reflect differences in the other characteristics of areas that receive support (e.g. rural or wealthier).

What did the evaluation do?

In order to address this issue, the study compared before-and-after changes in outcomes for supported areas to changes for unsupported areas. The study’s unit of observation is the zip code, with zip codes exposed to the loan programme serving as the ‘treatment’ group. Equally sparsely populated zip codes (i.e. with fewer than 20,000 residents) that were not exposed to the programme but were observably similar served as the ‘control’ group. The method the study uses to make this comparison is called a difference-in-differences model with propensity score matching.

How good was the evaluation?

According to our scoring guide, the difference-in-differences method receives a maximum of 3 (out of 5) on the Maryland Scientific Methods Scale. This is because it does well to control for observable (e.g. demographic characteristics) differences between treated and untreated areas, but is unable to control for time-variant unobservable differences (e.g. changes in institutional quality). In order to be well implemented, the control areas must be chosen so as to be as similar as possible to the treated areas. The study matches treated areas to untreated areas that are similar according to observable economic characteristics such as number of businesses and employment levels. Because the control group is similar to the treatment group, we score this study a 3 on the SMS.

What did the evaluation find?

The study found that the rollout did not have any effects, while the pilot had no impact on employment and annual payroll, but increased the number of establishments by 4.2 per cent. Note, however that the pilot programme had a positive and significant impact only in zip codes inside metropolitan areas. Adjacent and rural areas did not benefit appreciably from the loan programme. The mass roll-out of the policy had no effects on any outcomes nor in any areas, except for a negative impact on employment and payroll in rural counties.

What can we learn from this?

The study shows that the pilot programme was more effective in metropolitan areas. This implies that it may be more economically efficient to invest in underserved metropolitan areas that are perhaps overlooked because of low demand or purchasing power, rather than sparsely populated rural zones. The study also shows that while the mass rollout of the programme had no effect, the pilot programme yielded some positive outcomes. The authors speculate that this is due to the design of the pilot and rollout programmes. Since several companies defaulted on their loans in the pilot, the requirements for the rollout were much more stringent. As a result, the study suggests that the pilot programme was more similar to a grant programme. This could imply that grant programmes might be more effective than loans, particularly if there are areas in which companies would not otherwise invest.

Reference

Kandilov, I. T., & Renkow, M. (2010). Infrastructure investment and rural economic development: An evaluation of USDA’s broadband loan program. Growth and Change, 41(2), 165-191. [Study 769 from our Broadband review]

Downloads

Related Content

Register for WWG Updates