Now you're speaking my language: visual language identification

Conference item

The goal of this work is to train models that can identify a spoken language just by interpreting the speaker’s lip movements. Our contributions are the following: (i) we show that models can learn to discriminate among 14 different languages using only visual speech information; (ii) we compare different designs in sequence modelling and utterance-level aggregation in order to determine the best architecture for this task; (iii) we investigate the factors that contribute discriminative cues and show that our model indeed solves the problem by finding temporal patterns in mouth movements and not by exploiting spurious correlations. We demonstrate this further by evaluating our models on challenging examples from bilingual speakers.

Authors: Afouras, T; Chung, JS; Zisserman, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: ISCA Archive
• Host title: Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI
• Journal: Proceedings of ISCA 2020
• Issue: 2020
• Pages: 2402--2406
• Publication date: 2020-10-27
• Acceptance date: 2020-07-01
• Event title: ECCV
• Event location: Online
• Event website: https://eccv2020.eu/
• Event start date: 2020-08-23
• Event end date: 2020-08-28
• DOI: 10.21437/Interspeech.2020-2921
• ISSN: 1990-9772
• Language: English
• Keywords: FFR