Combine and conquer: representation learning from multmodal data

Thesis

Supervised learning, which involves training a model using a labelled dataset, is becoming less popular due to its high cost and various issues with generalization and robustness. This is unsurprising, as data such as images and language are complex and cannot be accurately represented by a single label. When models are trained using this method, they often learn features that spuriously correlate with the label, resulting in poor performance when deployed in the real world.

This thesis explores representation learning using multiple sources of data, such as images and language or photos and sketches. We demonstrate through both generative and discriminative models that extracting common abstract concepts between multiple modalities or domains can lead to more accurate and generalisable representations. In addition, we investigate ways to improve the data efficiency of these models, including using fewer multimodal pairs through contrastivestyle objectives and generating multimodal pairs through masked image modeling. Finally, we systematically evaluate the robustness of different learning objectives on distribution shift tasks in order to understand their usefulness in the real world.

Authors: Shi, Y

• DOI: 10.5287/ora-1zo2v9x1z
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: self-supervised learning; representation learning; multimodal data; unsupervised learning; generative model; robustness
• Subjects: Artificial intelligence