Prediction-powered machine learning for model selection and uncertainty

Thesis

This thesis explores model selection and uncertainty through the lens of prediction. Building on recent developments in Bayesian predictive inference, we approach uncertainty as a missing data problem, extending the logic of the bootstrap by treating future observations as the basis for inference. This framework offers a complementary perspective to conventional frequentist and Bayesian methods, as it supports probabilistic uncertainty quantification without requiring the subjective specification of a prior distribution. We first apply this lens to model uncertainty and hypothesis testing, proposing a novel procedure where uncertainty is propagated via the recursive imputation of new data, using a one-step-ahead model selection criterion. We then broaden this view, arguing that a model’s sequential predictive behavior — specifically, its production of conditionally identically distributed updates — can be used to characterize its coherence, allowing for Bayesian-style uncertainty even in plug-in or frequentist settings. Finally, we apply predictive model ensembling to causal treatment effect estimation, introducing a random forest method that targets relative risk heterogeneity, and demonstrating its application to data from a major cardiovascular clinical trial. Taken together, these results argue that predictive resampling methods, grounded in bootstrap principles, can provide flexible and principled tools for model evaluation and validation.

Authors: Shirvaikar, VM

• DOI: 10.5287/ora-j2o51ad0k
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: causal inference; predictive inference; model uncertainty; model selection; Bayesian statistics
• Subjects: Machine learning; Statistics