Geometry and representation learning in deep generative models

Thesis

Deep generative models have de facto emerged as state of the art when it comes to density estimation and sampling high-dimensional and multi-modal data. They combine the abstract yet practical mathematical description of probabilistic modelling with the flexibility and scalability brought by neural networks. They have been successfully applied to a wide spectrum of problems ranging from computer vision and natural language, to the realms of physical sciences. Probabilistic modelling, and in particular Bayesian statistics, have long enabled scientists and practitioners to include prior knowledge that they may have about the data into models. Building well-specified models is beneficial as it leads to improved generalisation capacity, data efficiency and interpretability of the model. In contrast, principled methods to encode such inductive biases in deep generative models are still under development.

This thesis presents three pieces of work aimed at addressing this problem. First, we propose a principled approach to introduce inductive biases in variational auto-encoders. Taking a Bayesian perspective, we show that encoding a desired structure into the prior distribution, and applying a proper regularisation, can lead to the desired decomposition in the learnt encodings. We demonstrate this approach on a variety of computer vision datasets and successfully learn representations with sparsity, clustering, and even intricate hierarchical dependency relationships. Next, we introduce an extension of variational auto-encoders to model data with underlying hierarchical structure. As hyperbolic spaces are perfectly suited to embed tree-like data, in contrast to Euclidean geometry, we endow the latent space with hyperbolic geometry. We do so by deriving the necessary methods to work with two main Gaussian generalisations and geometry-aware architectures for the encoder and decoder networks. Finally, we leverage the formalism of Riemannian geometry to define flexible distributions for data that are assumed to live on a given manifold. We do so by extending continuous normalising flows and parametrising manifold-valued diffeomorphisms as solutions of ordinary differential equations.

Authors: Mathieu, E

• DOI: 10.5287/ora-xmaaok6gz
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: deep learning; manifolds; geometry; machine learning; variational inference; probabilistic modelling
• Subjects: Deep learning (Machine learning); Machine learning; Manifolds (Mathematics); Statistics; Modeling; Inference; Geometry