On the inductive biases of deep generative models

Thesis

Generative modelling is an important machine learning paradigm for approximating high-dimensional distributions of multi-modal data. Generative probabilistic models learn a compressed latent representation and allow to sample from the approximated distribution, achieving state-of-the-art performance across numerous tasks. Many generative models feature latent variables which allow them to learn a hierarchical representation capturing structure in data or exploit its multi-resolution property. Others can be flexibly conditioned on available data at inference time. However, the inductive biases why generative models work well, and their critical components such as their neural architecture are often poorly understood.

In this thesis, we develop, characterise and probe the inductive biases of four major generative model classes: Variational Autoencoders (VAEs), Hierarchical VAEs (HVAEs), diffusion models, and autoregressive models or Large Language Models (LLMs). First, we propose Multi-Facet Clustering Variational Autoencoders (MFCVAE), a novel generative model which can uncover a multi-partition structure in data. Second, we identify state-of-the-art HVAEs as discretisations of an underlying multi-resolution diffusion process. Third, we analyse why U-Nets are a go-to architecture for the generative modelling of images and for diffusion models in particular, characterising them as learning the coefficients on truncated, finite-dimensional function spaces via preconditioning. Fourth, for LLMs, we analyse the hypothesis that in-context learning with i.i.d. data follows Bayesian principles. The goal of this thesis is to understand fundamental design principles and properties of deep generative models, which may characterise existing and inspire novel inductive biases.

Authors: Falck, F

• DOI: 10.5287/ora-wxjxe8d1d
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: PDE modelling; Bayesian inference; image generation; diffusion models; wavelets; machine learning; large language models; U-Net; ResNet; generative models; in-context learning; variational autoencoders; hierarchical variational autoencoders; multi-resolution analysis; clustering
• Subjects: Generative artificial intelligence; Machine learning; Neural networks (Computer science); Deep learning (Machine learning)