Generative models for generic data

Thesis

Generative models parameterized through large neural networks are now ubiquitous across machine learning. These models are able to generate highly realistic samples, ranging from human-like text to novel chemical compounds. With the ever growing list of potential application areas comes the need to continually adapt generative models to the complex data types that appear in the real world. This thesis aims to endow generative models with the ability to operate over generic data, requiring our generative frameworks to be general enough to handle any structures and regularities that may be present in the dataset. We focus on the subfamily of generative models that create data by simulating an iterative generative process which have become the best performing model class across a range of tasks due to their simple training procedure and high quality samples. This brings new challenges for generic data modelling because appropriate generative processes must be designed to operate on complex data spaces.

We begin by developing a method for generating discrete data through simulating continuous time Markov chains (CTMCs). Our approach translates techniques from continuous space diffusion models parameterized by stochastic differential equations into discrete space. We then turn our attention to data that can vary in dimensionality by giving continuous space diffusion models the ability to add dimensions as needed during generation. In the second half of the thesis, we switch focus to flow-based generative models which offer a simple alternative approach to creating generative processes. We first apply flows operating on the space of rigid body motions to the task of protein motif-scaffolding. We then expand flow-based modelling to discrete data and multi-modal data by utilizing CTMCs in combination with continuous space flows. We apply our method to protein co-design, simultaneously generating the continuous protein 3D structure and corresponding discrete amino acid sequence.

Authors: Campbell, A

• DOI: 10.5287/ora-prkjddoa7
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: flow matching; jump diffusion; discrete diffusion; protein design; generative models
• Subjects: Machine learning