CAT: enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios

Conference item

This paper focuses on the challenge of answering questions in scenarios that are composed of rich and complex dynamic audiovisual components. Although existing Multimodal Large Language Models (MLLMs) can respond to audio-visual content, these responses are sometimes ambiguous and fail to describe specific audio-visual events. To overcome this limitation, we introduce the CAT, which enhances MLLM in three ways: 1) besides straightforwardly bridging audio and video, we design a clue aggregator that aggregates question-related clues in dynamic audio-visual scenarios to enrich the detailed knowledge required for large language models. 2) CAT is trained on a mixed multimodal dataset, allowing direct application in audio-visual scenarios. Notably, we collect an audio-visual joint instruction dataset named AVinstruct, to further enhance the capacity of CAT to model cross-semantic correlations. 3) we propose AI-assisted ambiguity-aware direct preference optimization, a strategy specialized in retraining the model to favor the non-ambiguity response and improve the ability to localize specific audiovisual objects. Extensive experimental results demonstrate that CAT outperforms existing methods on multimodal tasks, especially in AudioVisual Question Answering (AVQA) tasks. The codes and the collected instructions are released at https://github.com/rikeilong/Bay-CAT

Authors: Ye, Q; Yu, Z; Shao, R; Xie, X; Torr, P

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Association for Computing Machinery
• Host title: Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part
• Pages: 146 - 164
• Publication date: 2024-11-03
• Acceptance date: 2024-02-26
• Event title: Computer Vision and Pattern Recognition (CVPR 2024)
• Event location: Seattle, WA, USA
• Event website: https://cvpr.thecvf.com/
• Event start date: 2024-06-17
• Event end date: 2024-06-21
• DOI: 10.1007/978-3-031-72684-2_9
• ISBN: 978-3-031-72683-5
• Language: English
• Keywords: multimodal large language model; audio-visual question answering