Synchformer: efficient synchronization from sparse cues

Conference item

Our objective is audio-visual synchronization with a focus on ‘in-the-wild’ videos, such as those on YouTube, where synchronization cues can be sparse. Our contributions include a novel audio-visual synchronization model, and training that decouples feature extraction from synchronization modelling through multi-modal segment-level contrastive pre-training. This approach achieves state-of-the-art performance in both dense and sparse settings. We also extend synchronization model training to AudioSet a million-scale ‘in-the-wild’ dataset, investigate evidence attribution techniques for interpretability, and explore a new capability for synchronization models: audio-visual synchronizability.

Authors: Iashin, V; Xie, W; Rahtu, E; Zisserman, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Host title: Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2024)
• Pages: 5325-5329
• Publication date: 2024-03-18
• Acceptance date: 2023-12-13
• Event title: International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2024)
• Event location: Seoul, Korea
• Event website: https://2024.ieeeicassp.org/
• Event start date: 2024-04-14
• Event end date: 2024-04-19
• DOI: 10.1109/ICASSP48485.2024.10448489
• EISSN: 2379-190X
• ISSN: 1520-6149
• EISBN: 9798350344851
• ISBN: 9798350344868
• Language: English
• Keywords: FFR; transformers; evidence attribution; multi-modal contrastive learning; audio-visual synchronization