Uncertainty estimation with small and large models

Thesis

Machine learning models have seen widespread use across many high-stakes domains, including healthcare and criminal justice. Because these models are used to inform important decisions, such as whether to offer a client a loan or not, it is critical to quantify uncertainty before acting. Without doing so, we risk taking inappropriate actions that place misguided trust in potentially flawed model estimates. Therefore, in this thesis, we develop approaches for uncertainty estimation across different domains. First, we focus on understanding the effects of government interventions on the transmission of COVID-19. We use Bayesian modelling, which provides a principled framework for inference and decision making under uncertainty. Specifically, we use semi-mechanistic hierarchical models to provide robust estimates of intervention effect sizes. In this setting, where model parameters and latent variables are semantically meaningful, datasets are small, and accurate inference is tractable, Bayesian methods excel. We then turn our attention towards supervised learning with neural networks. Unlike the COVID-19 models, approximate inference is inaccurate in this setting, even with large amounts of computational resources. Moreover, setting priors in these black-box models is challenging. To make progress, we argue algorithms for prediction need not maintain distributions over every model parameter and instead partially stochastic networks are equally well justified. We then develop partially stochastic Bayesian neural networks that leverage unlabelled data for improved prior predictive distributions. Following this, we show that Bayesian modelling can be fruitfully combined with modern unsupervised learning approaches by using large language models to produce features from structured inputs. These features can be fed into a Bayesian model to understand complex phenomena and provide uncertainty estimates. Overall, we show that several different approaches are needed for useful and appropriately uncertain predictions, and provide insight on the place of Bayesian methods in modern machine learning.

Authors: Sharma, M

• DOI: 10.5287/ora-d5mb9ro19
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: Bayesian; artificial intelligence; statistical modelling; uncertainty estimation; Bayesian neural network
• Subjects: Machine learning; Artificial Intelligence