Recognizing visual communication with limited supervision

Thesis

Understanding human communication is the first step towards enabling most human-machine interactions. Deep learning models of today can accurately transcribe and reason across spoken and written forms of human communication. In this work, we focus on the third frontier in human communication --- the visual language, specifically the signals communicated using our lips and hands.

In the first part of this work, we focus on transcribing lip movements to text. We develop new attention-based visual features to track the lip region in a dynamic talking head video. Next, we leverage recent advances in automatic speech recognition (ASR) and show that we can adapt a state-of-the-art ASR model for lip-reading. Finally, we extend lip reading to languages beyond English using a new large-scale multilingual dataset with weak ASR labels.

In the second part of this thesis, we propose Transpotter, a new architecture that can localize the word being uttered in the silent video. It outperforms previous methods by a large margin -- and demonstrates strong application in spotting mouthings in sign language videos.

In the third part of this work, we shift our attention to sign language and solve several problems in pursuit of our goal to recognize continuous signing sequences. We develop automatic methods to densely annotate a large corpora of signing videos with word labels. We also design a novel approach to recognize an important class of signs --- Fingerspelling. Finally, we move beyond word-level recognition to tackle continuous sign language recognition in large vocabulary settings.

In the final part of the thesis, we expand our area from signing gestures to co-speech gestures. We propose JEGAL, a joint embedding space to associate hand gestures to the words being uttered. We demonstrate the efficacy of these representations in strong downstream applications.

Authors: Kondajji Renukananda, P

• DOI: 10.5287/ora-gpeqqwn6p
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: sign language; lip reading; gesture understanding
• Subjects: Sign language; Speech and gesture; Lipreading; Deep learning (Machine learning); Computer vision