Probabilistic models and Monte Carlo objectives in generative modelling

Thesis

Generative modelling, i.e. learning probability distributions of interest, is a key and challenging problem in statistics and machine learning. With recent breakthroughs in deep learning, there have been remarkable advances in applying novel machine learning methods to solve generative modelling problems. In particular, with developments in variational inference, the latent variable model (LVM) has become a powerful class of generative models capable of modelling complex distributions flexibly. Recently, the denoising diffusion model (DDM) has emerged as another state-of-the-art generative model, which can also be viewed under the LVM framework but provides even higher expressivity and quality of learning and achieved exceptional results in many tasks such as unsupervised learning of images.

In this thesis, we present a variety of advances in theory and methodology for LVM and DDM, and explore a novel class of generative models named diffusion Schrödinger bridge (DSB). For LVMs, we propose unbiased multilevel Monte Carlo (MLMC) estimators of the log-likelihood and its gradients, as well as other quantities useful for learning the model and its variational posterior. For DDMs, we present a theoretical framework unifying their learning and usages in general data spaces. Finally, we focus on the recently developed DSB methodology which has close connections to DDMs as well as optimal transport (OT). We present a study on the application of DSB in the setting of conditional simulation and demonstrate its superior performance. We also present a novel algorithm for the learning of DSB models named DSBM, which achieves superior learning results as shown in problems such as image domain transfer and statistical downscaling.

Authors: Shi, Y

• DOI: 10.5287/ora-zbz7gzzmn
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: variational inference; generative modelling; Monte Carlo; denoising diffusion models; latent variable models
• Subjects: Machine learning; Statistics