Conference item icon

Conference item

Read and attend: temporal localisation in sign language videos

Abstract:
The objective of this work is to annotate sign instances across a broad vocabulary in continuous sign language. We train a Transformer model to ingest a continuous signing stream and output a sequence of written tokens on a large-scale collection of signing footage with weakly-aligned subtitles. We show that through this training it acquires the ability to attend to a large vocabulary of sign instances in the input sequence, enabling their localisation. Our contributions are as follows: (1) we demonstrate the ability to leverage large quantities of continuous signing videos with weakly-aligned subtitles to localise signs in continuous sign language; (2) we employ the learned attention to automatically generate hundreds of thousands of annotations for a large sign vocabulary; (3) we collect a set of 37K manually verified sign instances across a vocabulary of 950 sign classes to support our study of sign language recognition; (4) by training on the newly annotated data from our method, we outperform the prior state of the art on the BSL-1K sign language recognition benchmark.
Publication status:
Published
Peer review status:
Peer reviewed

Actions

Access Document

Files:
Publisher copy:
10.1109/CVPR46437.2021.01658

Authors

More by this author
Institution:
University of Oxford
Division:
MPLS
Department:
Engineering Science
Role:
Author
More by this author
Institution:
University of Oxford
Division:
MPLS
Department:
Engineering Science
Role:
Author
ORCID:
0000-0003-1732-9198
More by this author
Institution:
University of Oxford
Division:
MPLS
Department:
Engineering Science
Role:
Author
More by this author
Institution:
University of Oxford
Division:
MPLS
Department:
Engineering Science
Oxford college:
Brasenose College
Role:
Author
ORCID:
0000-0002-8945-8573


Royal Society More from this funder
Grant:
RP/R1/191132
Engineering and Physical Sciences Research Council More from this funder
Grant:
EP/R03298X/1


Publisher:
IEEE
Host title:
Proceedings of the 46th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2021)
Pages:
16852-16861
Publication date:
2021-11-13
Acceptance date:
2021-02-12
Event title:
46th IEEE International Conference on Acoustics, Speech and Signal Processing (CASSP 2021)
Event location:
Toronto, Canada
Event website:
https://2021.ieeeicassp.org/Papers.asp
Event start date:
2021-06-06
Event end date:
2021-06-11
DOI:
EISSN:
2575-7075
ISSN:
1063-6919
EISBN:
978-1-6654-4509-2
ISBN:
978-1-6654-4510-8


Language:
English
Keywords:
Pubs id:
1173944
Local pid:
pubs:1173944
Deposit date:
2021-04-28
ARK identifier:

Terms of use


Views and Downloads

Views and downloads will return soon






If you are the owner of this record, you can report an update to it here: Report update to this record

TO TOP