A course-focused dual curriculum for image captioning

Conference item

We propose a curriculum learning captioning method to caption fetal ultrasound images by training a model to dynamically transition between two different modalities (image and text) as training progresses. Specifically, we propose a course-focused dual curriculum method, where a course is training with a curriculum based on only one of the two modalities involved in image captioning. We compare two configurations of the course-focused dual curriculum; an image-first course-focused dual curriculum which prepares the early training batches primarily on the complexity of the image information before slowly introducing an order of batches for training based on the complexity of the text information, and a text-first course-focused dual curriculum which operates in reverse. The evaluation results show that dynamically transitioning between text and images over epochs of training improves results when compared to the scenario where both modalities are considered in equal measure in every epoch.

Authors: Alsharid, M; El-Bouri, R; Sharma, H; Drukker, L; Papageorghiou, AT

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Pages: 716-720
• Publication date: 2021-04-13
• Acceptance date: 2021-01-08
• Event title: 2021 IEEE 18th International Symposium on Biomedical Imaging
• Event location: Nice, France
• Event website: https://biomedicalimaging.org/2021/
• Event start date: 2021-04-13
• Event end date: 2021-04-16
• DOI: 10.1109/ISBI48211.2021.9434055
• EISSN: 1945-8452
• ISSN: 1945-7928
• EISBN: 978-1-6654-1246-9
• ISBN: 978-1-6654-2947-4
• Language: English
• Keywords: curriculum learning; fetal ultrasound; image captioning; FFR; image description; meta-learning