Using split samples to improve inference on causal effects

Journal article

We discuss a statistical procedure to carry out empirical research that combines recent insights about pre-analysis plans and replication. Researchers send their datasets to an independent third party who randomly generates training and testing samples. Researchers perform their analysis on the training sample and are able to incorporate feedback from both colleagues, editors and referees. Once the paper is accepted for publication the method is applied to the testing sample and it is those results that are published. Simulations indicate that, under empirically relevant settings, the proposed method delivers more power than a pre-analysis plan. The effect mostly operate through a lower likelihood that relevant hypotheses are left untested. The method appears better suited for exploratory analyses where there is significant uncertainty about the outcomes of interest. We do not recommend using the method in situations where the treatment are very costly and thus the available sample size is limited. An interpretation of the method is that it allows researchers to perform direct replication of their work. We also discuss a number of practical issues about the method’s feasibility and implementation.

Authors: Labonne, J; Fafchamps, M

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Oxford University Press
• Journal: Political Analysis
• Volume: 25
• Issue: 4
• Pages: 465-482
• Publication date: 2017-09-18
• Acceptance date: 2017-05-26
• DOI: 10.1017/pan.2017.22
• EISSN: 1476-4989
• ISSN: 1047-1987