Sample Survey Design for Impact Evaluation

The goal of impact evaluation is to estimate the causal relationship between input (control) variables and output (result) variables, such as the economic impact of a training program on farmer incomes or the health impact of a public-health program. This goal is achieved by means of causal modeling and analysis. There are two basic approaches to causal modeling and analysis. One is to set up an experimental design (randomized controlled trial) in which the treatment intervention is randomly assigned to program-eligible members of a population of interest. The other is to collect observational data on a population of interest, specify a causal model that shows the causal relationship among variables, and estimate the causal effect from the observational data, taking into account the causal model.

Estimation of causal impact involves use of the methodologies of experimental design and sample survey design. There are very large literatures on experimental design and descriptive sample survey design. In many applications, straightforward application of these methodologies is not practical or useful. For example, it may not be physically or ethically or politically practical to randomly assign treatment to subjects of interest. Descriptive sample survey design is the “usual” type of sample survey design. It is used to estimate characteristics of a population, with no consideration of impact estimation. It is used mainly in program monitoring, not in program evaluation. This type of survey design is of limited used in support of causal analysis, since, without randomized assignment of treatment, the observed treatment effect is usually a biased or inconsistent estimate of the average treatment effect. For causal analysis, what is required is an analytical survey design, which assures that the survey data will exhibit sufficient variation in important causal variables to produce unbiased or consistent impact estimates of useful precision.

In my statistical consulting practice, I specialize in the design of sample surveys in support of causal analysis. My approach to this topic is described in the article, Sample Survey Design for Evaluation (The Design of Analytical Surveys), posted at Internet website http://www.foundationwebsite.org/SampleSurveyDesignForEvaluation.pdf .

FndID(185)

FndTitle(Sample Survey Design for Impact Evaluation)

FndDescription(Sample Survey Design for Impact Evaluation)

FndKeywords(sample survey design; impact evaluation; program evaluation; statistics course; short course)