Visual grounding in video for unsupervised word translation

Journal article

There are thousands of actively spoken languages on Earth, but a single visual world. Grounding in this visual world has the potential to bridge the gap between all these languages. Our goal is to use visual grounding to improve unsupervised word mapping between languages. The key idea is to establish a common visual representation between two languages by learning embeddings from unpaired instructional videos narrated in the native language. Given this shared embedding we demonstrate that (i) we can map words between the languages, particularly the 'visual' words; (ii) that the shared embedding provides a good initialization for existing unsupervised text-based word translation techniques, forming the basis for our proposed hybrid visual-text mapping algorithm, MUVE; and (iii) our approach achieves superior performance by addressing the shortcomings of text-based methods - it is more robust, handles datasets with less commonality, and is applicable to low-resource languages. We apply these methods to translate words from English to French, Korean, and Japanese - all without any parallel corpora and simply by watching many videos of people speaking while doing things.

Authors: Sigurdsson, GA; Alayrac, JB; Nematzadeh, A; Smaira, L; Malinowski, M

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Journal: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
• Issue: 2020
• Pages: 10847-10856
• Publication date: 2020-08-05
• Acceptance date: 2020-02-27
• Event title: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• Event location: Online
• Event website: http://cvpr2020.thecvf.com/
• Event start date: 2020-06-14
• Event end date: 2020-06-19
• DOI: 10.1109/CVPR42600.2020.01086
• EISSN: 2575-7075
• ISSN: 1063-6919
• EISBN: 978-1-7281-7168-5
• ISBN: 978-1-7281-7169-2
• Language: English
• Keywords: FFR