Conference item
Weakly supervised captioning of ultrasound images
- Abstract:
- Medical image captioning models generate text to describe the semantic contents of an image, aiding the non-experts in understanding and interpretation. We propose a weakly-supervised approach to improve the performance of image captioning models on small image-text datasets by leveraging a large anatomically-labelled image classification dataset. Our method generates pseudo-captions (weak labels) for caption-less but anatomically-labelled (class-labelled) images using an encoder-decoder sequence-to-sequence model. The augmented dataset is used to train an image-captioning model in a weakly supervised learning manner. For fetal ultrasound, we demonstrate that the proposed augmentation approach outperforms the baseline on semantics and syntax-based metrics, with nearly twice as much improvement in value on BLEU-1 and ROUGE-L. Moreover, we observe that superior models are trained with the proposed data augmentation, when compared with the existing regularization techniques. This work allows seamless automatic annotation of images that lack human-prepared descriptive captions for training image-captioning models. Using pseudo-captions in the training data is particularly useful for medical image captioning when significant time and effort of medical experts is required to obtain real image captions.
- Publication status:
- Published
- Peer review status:
- Peer reviewed
Actions
Access Document
- Files:
-
-
(Preview, Accepted manuscript, pdf, 1.4MB, Terms of use)
-
- Publisher copy:
- 10.1007/978-3-031-12053-4_14
Authors
- Publisher:
- Springer
- Pages:
- 187-198
- Series:
- Lecture Notes in Computer Science
- Series number:
- 13413
- Publication date:
- 2022-07-25
- Event title:
- 26th Medical Image Understanding and Analysis (MUIA 2022)
- Event location:
- Cambridge, UK
- Event website:
- https://www.miua2022.com/
- Event start date:
- 2022-07-27
- Event end date:
- 2022-07-29
- DOI:
- EISSN:
-
1611-3349
- ISSN:
-
0302-9743
- ISBN:
- 9783031120527
- Language:
-
English
- Keywords:
- Pubs id:
-
1278280
- Local pid:
-
pubs:1278280
- Deposit date:
-
2023-02-21
- ARK identifier:
Terms of use
- Copyright holder:
- Alsharid et al
- Copyright date:
- 2022
- Rights statement:
- © 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
- Notes:
- This paper was presented at the 26th Medical Image Understanding and Analysis (MUIA 2022), 27th - 29th July 2022, Cambridge, UK. This is the accepted manuscript version of the article. The final version is available online from Springer at: https://doi.org/10.1007/978-3-031-12053-4_14
If you are the owner of this record, you can report an update to it here: Report update to this record