Multi-modal learning from video, eye tracking, and pupillometry for operator skill characterization in clinical fetal ultrasound

Conference item

This paper presents a novel multi-modal learning approach for automated skill characterization of obstetric ultrasound operators using heterogeneous spatio-temporal sensory cues, namely, scan video, eye-tracking data, and pupillometric data, acquired in the clinical environment. We address pertinent challenges such as combining heterogeneous, small-scale and variable-length sequential datasets, to learn deep convolutional neural networks in real-world scenarios. We propose spatial encoding for multi-modal analysis using sonography standard plane images, spatial gaze maps, gaze trajectory images, and pupillary response images. We present and compare five multi-modal learning network architectures using late, intermediate, hybrid, and tensor fusion. We build models for the Heart and the Brain scanning tasks, and performance evaluation suggests that multi-modal learning networks outperform uni-modal networks, with the best-performing model achieving accuracies of 82.4% (Brain task) and 76.4% (Heart task) for the operator skill classification problem.

Authors: Sharma, H; Drukker, L; Papageorghiou, AT; Noble, JA

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Pages: 1646-1649
• Publication date: 2021-04-13
• Acceptance date: 2021-01-08
• Event title: 2021 IEEE 18th International Symposium on Biomedical Imaging
• Event location: Nice, France
• Event website: https://biomedicalimaging.org/2021/
• Event start date: 2021-04-13
• Event end date: 2021-04-16
• DOI: 10.1109/ISBI48211.2021.9433863
• EISSN: 1945-8452
• ISSN: 1945-7928
• EISBN: 978-1-6654-1246-9
• ISBN: 978-1-6654-2947-4
• Language: English
• Keywords: FFR; eye tracking; convolutional neural networks; multi-modal learning; pupillometry; ultrasound