Conference item
Objects that sound
- Abstract:
-
In this paper our objectives are, first, networks that can embed audio and visual inputs into a common space that is suitable for cross-modal retrieval; and second, a network that can localize the object that sounds in an image, given the audio signal. We achieve both these objectives by training from unlabelled video using only audio-visual correspondence (AVC) as the objective function. This is a form of cross-modal self-supervision from video.
To this end, we design new network architectures that can be trained for cross-modal retrieval and localizing the sound source in an image, by using the AVC task. We make the following contributions: (i) show that audio and visual embeddings can be learnt that enable both within-mode (e.g. audio-to-audio) and between-mode retrieval; (ii) explore various architectures for the AVC task, including those for the visual stream that ingest a single image, or multiple images, or a single image and multi-frame optical flow; (iii) show that the semantic object that sounds within an image can be localized (using only the sound, no motion or flow information); and (iv) give a cautionary tale on how to avoid undesirable shortcuts in the data preparation.
- Publication status:
- Published
- Peer review status:
- Peer reviewed
Actions
Access Document
- Files:
-
-
(Preview, Accepted manuscript, pdf, 2.6MB, Terms of use)
-
- Publisher copy:
- 10.1007/978-3-030-01246-5_27
Authors
- Publisher:
- Springer
- Host title:
- Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I
- Pages:
- 451-466
- Series:
- Lecture Notes in Computer Science
- Series number:
- 11205
- Place of publication:
- Cham, Switerland
- Publication date:
- 2018-10-06
- Acceptance date:
- 2018-07-03
- Event title:
- 15th European Conference on Computer Vision (ECCV 2018)
- Event location:
- Munich, Germany
- Event website:
- https://eccv2018.org/
- Event start date:
- 2018-09-08
- Event end date:
- 2018-09-14
- DOI:
- EISSN:
-
1611-3349
- ISSN:
-
0302-9743
- EISBN:
- 9783030012465
- ISBN:
- 9783030012458
- Language:
-
English
- Keywords:
- Pubs id:
-
pubs:966740
- UUID:
-
uuid:d0bf6ced-8ccb-4edf-8ec0-02ad2d06efe8
- Local pid:
-
pubs:966740
- Source identifiers:
-
966740
- Deposit date:
-
2019-01-29
- ARK identifier:
Terms of use
- Copyright holder:
- Springer Nature Switzerland AG
- Copyright date:
- 2018
- Rights statement:
- © 2018 Springer Nature Switzerland AG.
- Notes:
- This is the accepted manuscript version of the paper. The final version is available online from Springer at https://dx.doi.org/10.1007/978-3-030-01246-5_27
If you are the owner of this record, you can report an update to it here: Report update to this record