What Have We Done? Impact Analysis and Program Evaluation

If policy analysis is an attempt to answer the question of “what should we do?”, program evaluation is a search for the answer to the question of “what have we done?” This chapter focuses heavily on impact analysis, which is a primary approach used in both academic and applied policy analysis. Impact analysis is squarely in the rationalist tradition, and the same post-positivist criticisms that are launched against policy analysis are also made against impact analysis. Impact analysis is subject to a broad set of objections from post-positivist critics. Again, these criticisms have some merit. An impact analysis is wholly dependent on generating a valid estimate of the counterfactual, and this is possible only if the correct activity and outcome of interest have been identified and accurately measured, the causal link connecting them is accurately understood, and these elements are brought together in a robust and well-thought-out research design.