Extending probabilistic programming systems and applying them to real-world simulators

Thesis

Probabilistic programming is a paradigm that enables us to efficiently write probabilistic models as program code that we can sample, infer underlying parameters and predict outcomes based on complete or incomplete observations. Naturally, stochastic simulators, a special sub-class of simulators containing random variables, internal inference procedures, and the simulation of observations, are structurally rich probabilistic models. However, most simulators are not written in the probabilistic programming paradigm, as they are written in arbitrary programming code. This means that it is challenging to automatically update the variables in these simulators to account for observations from conducted experiments, which limits the simulators' use.

Furthermore, there are two components to a probabilistic programming system i) the language and compilation procedure, ii) the inference procedures. These components can limit our ability to compile particular classes of probabilistic models, such as models that contain mixtures of parameter types, due to restrictions in the expressiveness of the language. Restrictions in the expressivity of the language can also inhibit our ability to generate efficient inferences, as this naturally influences the design of the probabilistic programming system and the set of available inference backends. Creating probabilistic programming systems that are expressive enough for different probabilistic models leads to the creation of many different probabilistic programming systems, which is inefficient - it would be more efficient if we could repurpose existing probabilistic programming systems.

In this thesis, we develop three pieces of original work through four papers. The first piece of work describes how to extend differentiable first-order probabilistic programming systems to perform statistically correct and computationally efficient inference on models with mixtures of continuous and non-continuous parameters, without having to modify the underlying language, or develop an entirely new probabilistic programming system. The second describes how to translate real-world stochastic simulators written in arbitrary program languages to probabilistic programming systems. And finally, in the third piece of work, we develop two new Bayesian inference schemes to make inference more computationally and statistically efficient in nested models, models that contain probabilistic programs, within probabilistic programs, which arise in many real-world stochastic simulators.

Authors: Gram-Hansen, B

• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: machine learning; simulators; probabilistic programming; statistics; Bayesian inference; AI
• Subjects: Statistics; Machine learning; Inference; Artificial intelligence