Conference item
A sound approach: using large language models to generate audio descriptions for egocentric text-audio retrieval
- Abstract:
- Video databases from the internet are a valuable source of text-audio retrieval datasets. However, given that sound and vision streams represent different "views" of the data, treating visual descriptions as audio descriptions is far from optimal. Even if audio class labels are present, they commonly are not very detailed, making them unsuited for text-audio retrieval. To exploit relevant audio information from video-text datasets, we introduce a methodology for generating audio-centric descriptions using Large Language Models (LLMs). In this work, we consider the egocentric video setting and propose three new text-audio retrieval benchmarks based on the EpicMIR and EgoMCQ tasks, and on the EpicSounds dataset. Our approach for obtaining audio-centric descriptions gives significantly higher zero-shot performance than using the original visual-centric descriptions. Furthermore, we show that using the same prompts, we can successfully employ LLMs to improve the retrieval on EpicSounds, compared to using the original audio class labels of the dataset. Finally, we confirm that LLMs can be used to determine the difficulty of identifying the action associated with a sound.
- Publication status:
- Published
- Peer review status:
- Peer reviewed
Actions
Access Document
- Files:
-
-
(Preview, Accepted manuscript, pdf, 5.7MB, Terms of use)
-
- Publisher copy:
- 10.1109/icassp48485.2024.10448486
Authors
- Publisher:
- IEEE
- Host title:
- Proceedings of the 49th IEEE International Conference on Acoustics, Speech, & Signal Processing (ICASSP 2024)
- Pages:
- 7300-7304
- Publication date:
- 2024-04-14
- Acceptance date:
- 2024-04-14
- Event title:
- 49th IEEE International Conference on Acoustics, Speech, & Signal Processing (ICASSP 2024)
- Event location:
- Seoul, South Korea
- Event website:
- https://2024.ieeeicassp.org/
- Event start date:
- 2024-04-14
- Event end date:
- 2024-04-19
- DOI:
- EISSN:
-
2379-190X
- ISSN:
-
1520-6149
- EISBN:
- 979-8-3503-4485-1
- ISBN:
- 979-8-3503-4486-8
- Language:
-
English
- Keywords:
- Pubs id:
-
1987486
- Local pid:
-
pubs:1987486
- Deposit date:
-
2024-06-12
- ARK identifier:
Terms of use
- Copyright holder:
- IEEE
- Copyright date:
- 2024
- Rights statement:
- © 2024 IEEE
- Notes:
- This paper was presented at the 49th IEEE International Conference on Acoustics, Speech, & Signal Processing (ICASSP 2024), 14th-19th April 2024, Seoul, South Korea. This is the accepted manuscript version of the article. The final version is available online from IEEE at: https://dx.doi.org/10.1109/icassp48485.2024.10448486
If you are the owner of this record, you can report an update to it here: Report update to this record