Enhancing experimental design through Bayes factor design analysis: insights from multi-armed bandit tasks

Journal article

Bayesian statistics offers a flexible framework that supports iterative updating of hypotheses and the incorporation of prior information, amongst other advantages. Although well established for retrospective analysis, the application of Bayesian methods to prospective analysis is less well developed, especially when used in combination with computational model-based analysis of behavioural data in cognitive neuroscience. It is therefore important to establish effective methods for testing and optimising experimental designs for these purposes. One framework for a prospective approach is Bayes factor design analysis (BFDA), which can be used alongside latent variable modelling to evaluate and visualise the distribution of Bayes factors for a given experimental design. This paper provides a tutorial-style analysis combining BFDA with latent variable modelling to evaluate exploration-exploitation trade-offs in the binary multi-armed bandit task (MAB). This is a complex example of human decision-making with which to investigate the feasibility of differentiating latent variables between groups as a function of different design parameters. We examined how sample size, number of games per participant and effect size affect the strength of evidence supporting a difference in means between two groups. To further assess how these parameters affect experimental results, metrics of error were evaluated. Using simulations, we demonstrated how BFDA can be combined with latent variable modelling to evaluate and optimise parameter estimation of exploration in the MAB task, allowing inference of the mean degree of random exploration in a population, as well as between groups. However, BFDA indicated that, even with large samples and effect sizes, there may be some circumstances where there is a high likelihood of errors and a low probability of detecting evidence in favour of a difference when comparing random exploration between two groups performing the bandit task. In summary, we show how BFDA can prospectively inform design and power of human behavioural tasks.

Authors: Schreiber, S; Hewitt, D; Seymour, B; Yoshida, W

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Taylor and Francis
• Journal: Wellcome Open Research
• Volume: 9
• Pages: 423
• Publication date: 2024-01-01
• Acceptance date: 2026-04-29
• DOI: 10.12688/wellcomeopenres.22288.2
• EISSN: 2398502X
• ISSN: 2398502X
• Pmid: 42256493
• Language: English
• Keywords: Design Analysis; Bayes Factor; Sample size; decision-making; Design Calculation; Exploration; Statistical Evidence