Shot-by-shot: film-grammar-aware training-free audio description generation

Conference item

Our objective is the automatic generation of Audio Descriptions (ADs) for edited video material, such as movies and TV series. To achieve this, we propose a two-stage framework that leverages “shots” as the fundamental units of video understanding. This includes extending temporal context to neighbouring shots and incorporating film grammar devices, such as shot scales and thread structures, to guide AD generation. Our method is compatible with both open-source and proprietary VisualLanguage Models (VLMs), integrating expert knowledge from add-on modules without requiring additional training of the VLMs. We achieve state-of-the-art performance among all prior training-free approaches and even surpass fine-tuned methods on several benchmarks. To evaluate the quality of predicted ADs, we introduce a new evaluation measure – an action score – specifically targeted to assessing this important aspect of AD. Additionally, we propose a novel evaluation protocol that treats automatic frameworks as AD generation assistants and asks them to generate multiple candidate ADs for selection.

Authors: Xie, J; Han, T; Bain, M; Nagrani, A; Khandelwal, E

• Publication status: Accepted
• Peer review status: Peer reviewed
• Publisher: IEEE
• Acceptance date: 2025-07-23
• Event title: International Conference on Computer Vision (ICCV 2025)
• Event location: Honolulu, Hawai'i, USA
• Event website: https://iccv.thecvf.com/
• Event start date: 2025-10-19
• Event end date: 2025-10-23
• Language: English