Captioning ultrasound images automatically

Conference item

We describe an automatic natural language processing (NLP)-based image captioning method to describe fetal ultrasound video content by modelling the vocabulary commonly used by sonographers and sonologists. The generated captions are similar to the words spoken by a sonographer when describing the scan experience in terms of visual content and performed scanning actions. Using full-length second-trimester fetal ultrasound videos and text derived from accompanying expert voice-over audio recordings, we train deep learning models consisting of convolutional neural networks and recurrent neural networks in merged configurations to generate captions for ultrasound video frames. We evaluate different model architectures using established general metrics (BLEU, ROUGE-L) and application-specific metrics. Results show that the proposed models can learn joint representations of image and text to generate relevant and descriptive captions for anatomies, such as the spine, the abdomen, the heart, and the head, in clinical fetal ultrasound scans.

Authors: Alsharid, M; Sharma. H; Drukker, L; Chatelain, P; Papageorghiou, AT

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Springer, Cham
• Host title: MICCAI 2019: Medical Image Computing and Computer Assisted Intervention – MICCAI 2019
• Pages: 338-346
• Series: Lecture Notes in Computer Science
• Series number: 11767
• Publication date: 2019-10-10
• Acceptance date: 2019-06-23
• Event title: Medical Image Computing and Computer Assisted Intervention – MICCAI 2019
• Event location: Germany
• DOI: 10.1007/978-3-030-32251-9_37
• EISSN: 1611-3349
• ISSN: 0302-9743
• EISBN: 978-3-030-32251-9
• ISBN: 9783030322502
• Language: English
• Keywords: natural language processing; fetal ultrasound; deep learning; image description; image captioning; recurrent neural networks