Learnable PINs: Cross-modal embeddings for person identity

Conference item

We propose and investigate an identity sensitive joint embedding of face and voice. Such an embedding enables cross-modal retrieval from voice to face and from face to voice. We make the following four contributions: first, we show that the embedding can be learnt from videos of talking faces, without requiring any identity labels, using a form of cross-modal self-supervision; second, we develop a curriculum learning schedule for hard negative mining targeted to this task that is essential for learning to proceed successfully; third, we demonstrate and evaluate cross-modal retrieval for identities unseen and unheard during training over a number of scenarios and establish a benchmark for this novel task; finally, we show an application of using the joint embedding for automatically retrieving and labelling characters in TV dramas.

Authors: Nagrani, A; Albanie, S; Zisserman, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Springer
• Host title: Lecture Notes in Computer Science
• Journal: Lecture Notes in Computer Science
• Volume: 11217
• Pages: 73-89
• Publication date: 2018-10-06
• Acceptance date: 2018-07-03
• DOI: 10.1007/978-3-030-01261-8_5
• ISSN: 1611-3349 and 0302-9743
• ISBN: 9783030012601
• Keywords: cross-modal; face recognition; self-supervised; joint embedding; speaker identification; metric learning; multi-modal