Rationalizing pre-analysis plans: statistical decisions subject to implementability

Working paper

Pre-analysis plans (PAPs) are a potential remedy to the publication of spurious findings in empirical research, but they have been criticized for their costs and for preventing valid discoveries. In this article, we analyze the costs and benefits of pre-analysis plans by casting pre-commitment in empirical research as a mechanism-design problem. In our model, a decision-maker commits to a decision rule. Then an analyst chooses a PAP, observes data, and reports selected statistics to the decision-maker, who applies the decision rule. With conflicts of interest and private information, not all decision rules are implementable. We provide characterizations of implementable decision rules, where PAPs are optimal when there are many analyst degrees of freedom and high communication costs. These PAPs improve welfare by enlarging the space of implementable decision functions. This stands in contrast to single-agent statistical decision theory, where commitment devices are unnecessary if preferences are consistent across time.

Authors: Kasy, M; Spiess, J

• Publication status: Published
• Publisher: University of Oxford
• Series: Department of Economics Discussion Paper Series
• Place of publication: Oxford
• Publication date: 2022-06-21
• ISSN: 1471-0498
• Paper number: 975
• Language: English
• Keywords: pre-analysis plans; statistical decisions; implementability