Localizing events in videos with multimodal queries

Conference item

Localizing events in videos based on semantic queries is a pivotal task in video understanding research and user-oriented applications like video search. Yet, current research predominantly relies on natural language queries (NLQs), overlooking the potential of using multimodal queries (MQs) that incorporate images to flexibly represent semantic queries, particularly when it is difficult to express non-verbal or unfamiliar concepts in words. To bridge this gap, we introduce ICQ, a new benchmark designed for localizing events in videos with MQs, alongside an evaluation dataset ICQ-Highlight. To adapt and reevaluate existing video localization models for this new task, we propose 3 Multimodal Query Adaptation methods and a novel Surrogate Fine-Tuning strategy, serving as strong baseline methods. ICQ systematically benchmarks 12 state-of-the-art backbone models, spanning from specialized video localization models to Video Large Language Models. Our extensive experiments highlight the high potential of using MQs in real-world applications. We believe this is a first step toward video event localization with MQs.

Authors: Zhang, G; Fok, MLA; Ma, J; Xia, Y; Cremers, D

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Host title: 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• Pages: 3339-3351
• Publication date: 2025-06-10
• Acceptance date: 2025-02-27
• Event title: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2025)
• Event location: Nashville, TN, USA
• Event website: https://cvpr.thecvf.com/
• Event start date: 2025-06-11
• Event end date: 2025-06-15
• DOI: 10.1109/CVPR52734.2025.00317
• EISSN: 2575-7075
• ISSN: 1063-6919
• EISBN: 9798331543648
• ISBN: 9798331543655
• Language: English
• Keywords: video understanding; temporal grounding; video grounding; multimodal large language model