Multimodal learning with transformers: a survey

Journal article

Transformer is a promising neural network learner, and has achieved great success in various machine learning tasks. Thanks to the recent prevalence of multimodal applications and Big Data, Transformer-based multimodal learning has become a hot topic in AI research. This paper presents a comprehensive survey of Transformer techniques oriented at multimodal data. The main contents of this survey include: (1) a background of multimodal learning, Transformer ecosystem, and the multimodal Big Data era, (2) a systematic review of <italic>Vanilla</italic> Transformer, Vision Transformer, and multimodal Transformers, from a geometrically topological perspective, (3) a review of multimodal Transformer applications, via two important paradigms, <italic>i</italic>.<italic>e</italic>., for multimodal pretraining and for specific multimodal tasks, (4) a summary of the common challenges and designs shared by the multimodal Transformer models and applications, and (5) a discussion of open problems and potential research directions for the community.

Authors: Xu, P; Zhu, X; Clifton, DA

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Journal: IEEE Transactions on Pattern Analysis and Machine Intelligence
• Volume: 45
• Issue: 10
• Pages: 12113-12132
• Publication date: 2023-05-11
• DOI: 10.1109/TPAMI.2023.3275156
• EISSN: 1939-3539
• ISSN: 0162-8828
• Language: English
• Keywords: FFR