Econometric methods of impact evaluation

Data-based impact evaluation methods are nowadays frequently used to measure the effect of public interventions (reforms, assistance programs, etc.). It is important to distinguish ex ante evaluations, which are based on the simulation of the envisioned intervention based on a structural econometric model estimated before its implementation (see below), and ex post evaluations, which are implemented using data collected after its implementation.

Essentially, an ex post evaluation is based on a comparison between individuals, households or firms benefiting from the intervention (the treatment group) and individuals, households or companies not benefiting from it (the control group, or counterfactual). Different econometric methods exist, and the choice of the right approach strongly depends on how the control group and the treatment group were formed. The ideal framework of analysis is a controlled experiment where the units in these two groups are randomly selected. This situation is, however, relatively rare in practice.

In the absence of such random selection, estimates are likely to be polluted by a selection bias in these two groups. When this selection bias can be explained by observable variables (under the hypothesis of independence conditional on observable characteristics), it is possible to use classical regression techniques to estimate the impact of reforms. Other methods, such as matching or propensity score, are also possible.

When the conditional independence assumption is violated, these methods produce biased estimates. It is then necessary to consider other approaches:

Instrumental variables
This method uses variables correlated with the treatment variable, but not with the variable measuring the outcome of the reform.

Regression discontinuity design
This method is used in situations where the probability of treatment is a discontinuous function of the value taken by a covariate other than the outcome variable (for example, the age or income of the individual or the size of the business).

The double differences estimator (or difference in differences)
This widely-used method consists of contrasting the change in the outcome variable in the control group with that in the treatment group, before and after the intervention. Its main weakness lies in the hypothesis of “parallel evolution” of control and treatment groups in the absence of public intervention.

Synthetic Control Method (SCM)
This is a more recent method that makes it possible to dispense with the hypothesis of parallel evolution. The SCM creates a synthetic control unit that represents the counterfactual of the “treated” unit (the unit benefiting from the intervention). This synthetic unit is created automatically by applying optimal weights to all candidate control units so as to minimize the distance (before intervention) between the treated unit and the synthetic control unit. An example of application is provided below.

Application example: Synthetic Control Method

The SCM was recently applied by ECOPA to evaluate a series of reforms in the countries of the Organization for Harmonization of Business Law in Africa (OHADA). An example of these reforms is the Uniform Security Organization Act, implemented in 2011, which broadened the range of assets that can be used as collateral and introduced the realization of an “autonomous” and extrajudicial security. The SCM analysis has made it possible to rigorously estimate the impact of this reform on access to credit in 10 member countries of the OHADA. The results indicate that 7 of the 10 countries (Burkina Faso, Cameroon, Comoros, Mali, Senegal, Togo and the Central African Republic) have seen a strong positive impact in terms of additional credits, ranging from USD 30 million (Comoros) to more than USD 1 billion (Senegal) over the period 2011-2015.

Analysis of time series and panel data

Econometric analysis of time series can allow ex ante evaluations based on relatively simple data. This consists of estimating the correlations between several time series (for example, time series of the tax rates of a given tax and the corresponding revenue) and deducing from it the probable response of a variable of interest as a function of an exogenous change in a policy variable (e.g., revenue response to a change of a given tax rate).

This approach uses econometric techniques to hold constant foreign exogenous effects on the policy or program to be evaluated. For example, in taxation analysis, the effect of a tax rate on the corresponding tax revenue can be influenced by GDP per capita. It is then introduced into the econometric model as a control variable, which isolates the effect of the tax rate.

Panel data analysis is an extension of time series analysis where data includes several observations over time for each individual (household, firm, region, etc.) in the population.

These techniques are usually simple from the point of view of the causal mechanism tested, but require solid econometric skills to adapt the approach to the structure of the data.

Micro simulations

Micro simulation models are analytical tools for simulating the effects of a given reform or program on a sample of agents (individuals or firms), or the entire population where possible, at the individual level.

For example, in tax policy analysis, a micro-simulation model can be effective in simulating the ex ante effects of a change in the progressive personal income tax schedule by applying it individually to each taxpayer so as to infer the new distribution of income tax due and the total expected income. Similarly, the micro simulation approach has been widely used to simulate the effects of household support policies based on their individual characteristics.

While the theory behind micro simulation models is generally simple, the analytical challenge lies in the preparation and manipulation of often cumbersome microeconomic data.

ECOPA has frequently implemented micro simulation models in the field of business and household taxation.

Macroeconomic models of computable general equilibrium

These models are based on national accounts tracing exchanges between agents and sectors of a region or country. They allow for the ex ante evaluation of a policy shock, such as a change in the rate of a given tax instrument or a reduction in tariffs as a result of a free trade agreement.

These models, which are implemented in specialized software (e.g. GAMS), are powerful but complex. Their implementation has a high cost of entry, especially as regards the development of the information base adapted to the policy or program to be evaluated. For example, for a measure supporting households, it will be necessary to isolate in the data the households concerned and the flow of income that will be directly affected by the support program concerned. These techniques should therefore be reserved for large-scale evaluations. They have been proven as very efficient in the ex ante analysis of international trade agreements.