Cased-based modelling and scenario simulation for ex-post evaluation

Journal article

Despite 20 years of increasing acceptance, implementing complexity-appropriate methods for ex-post evaluation remains a challenge: instead of focusing on complex interventions, methods need to help evaluators better explore how policies (no matter how simple) take place in real-world, open, dynamic systems where many intertwined factors about the cases being targeted affect outcomes in numerous ways. To assist in this advance, we developed case-based scenario simulation, a new visually intuitive evaluation tool grounded in a data-driven, case-based, computational modelling approach, which evaluators can use to explore counterfactuals, status-quo trends, and what-if scenarios for some potential set of real or imagined interventions. To demonstrate the value and versatility of case-based scenario simulation we explore four published evaluations that differ in design (cross sectional, longitudinal, and experimental) and purpose (learning or accountability), and present a prospective view of how case-based scenario simulation could support and enhance evaluators’ efforts in these complex contexts.

Authors: Schimpf, C; Barbrook-Johnson, P; Castellani, B

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: SAGE Publications
• Journal: Evaluation
• Volume: 27
• Issue: 1
• Pages: 116-137
• Publication date: 2021-01-14
• DOI: 10.1177/1356389020978490
• EISSN: 1461-7153
• ISSN: 1356-3890
• Language: English
• Keywords: case-based methods; FFR; scenario analysis; evaluation; computational social science; policy; social complexity