Conference item
Read and attend: temporal localisation in sign language videos
- Abstract:
- The objective of this work is to annotate sign instances across a broad vocabulary in continuous sign language. We train a Transformer model to ingest a continuous signing stream and output a sequence of written tokens on a large-scale collection of signing footage with weakly-aligned subtitles. We show that through this training it acquires the ability to attend to a large vocabulary of sign instances in the input sequence, enabling their localisation. Our contributions are as follows: (1) we demonstrate the ability to leverage large quantities of continuous signing videos with weakly-aligned subtitles to localise signs in continuous sign language; (2) we employ the learned attention to automatically generate hundreds of thousands of annotations for a large sign vocabulary; (3) we collect a set of 37K manually verified sign instances across a vocabulary of 950 sign classes to support our study of sign language recognition; (4) by training on the newly annotated data from our method, we outperform the prior state of the art on the BSL-1K sign language recognition benchmark.
- Publication status:
- Published
- Peer review status:
- Peer reviewed
Actions
Access Document
- Files:
-
-
(Preview, Accepted manuscript, pdf, 2.5MB, Terms of use)
-
- Publisher copy:
- 10.1109/CVPR46437.2021.01658
Authors
- Publisher:
- IEEE
- Host title:
- Proceedings of the 46th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2021)
- Pages:
- 16852-16861
- Publication date:
- 2021-11-13
- Acceptance date:
- 2021-02-12
- Event title:
- 46th IEEE International Conference on Acoustics, Speech and Signal Processing (CASSP 2021)
- Event location:
- Toronto, Canada
- Event website:
- https://2021.ieeeicassp.org/Papers.asp
- Event start date:
- 2021-06-06
- Event end date:
- 2021-06-11
- DOI:
- EISSN:
-
2575-7075
- ISSN:
-
1063-6919
- EISBN:
- 978-1-6654-4509-2
- ISBN:
- 978-1-6654-4510-8
- Language:
-
English
- Keywords:
- Pubs id:
-
1173944
- Local pid:
-
pubs:1173944
- Deposit date:
-
2021-04-28
- ARK identifier:
Terms of use
- Copyright holder:
- IEEE
- Copyright date:
- 2021
- Rights statement:
- © IEEE 2021.
- Notes:
- This paper was presented at the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2021), 6th-11th June 2021, Toronto, Canada. This is the accepted manuscript version of the article. The final version is available online from IEEE at: https://doi.org/10.1109/CVPR46437.2021.01658
If you are the owner of this record, you can report an update to it here: Report update to this record