Experimental evaluation of multimodal machine learning models for cardiovascular conditions in ICU patients

Thesis

Intensive care units (ICUs) generate vast amounts of heterogeneous data from various sources, ranging from imaging and waveforms to tabular records. Whereas clinicians routinely synthesise these diverse sources of information to guide patient care, traditional automated clinical decisionsupport systems have rarely integrated data across multimodal channels. Machine learning has the potential to replicate this synthesis through multimodal approaches, though the conditions under which accuracy genuinely improves remain unclear.

However, the development of such systems has been constrained by a lack of accessible, linked datasets and a limited understanding of data fusion methodologies. Consequently, how these data modalities interact remains largely unexplored. Specifically, there is a lack of systematic understanding regarding the specific conditions under which multimodal machine learning improves predictive performance, rather than simply introducing redundancy or algorithmic shortcuts.

To address this gap, this thesis establishes a framework to evaluate the performance of different methods for combining information from heterogeneous data sources, including imaging (X-rays), waveforms (ECGs), text (clinical notes) and tabular data (lab tests). The primary objective is to determine whether such integration improves model accuracy in an ICU setting and under which specific conditions it improves performance. Answering these questions became possible only recently following the accessibility of the MIMIC-IV database in 2021 and its linkage to ECG waveforms in 2023. Using cardiovascular conditions, a leading cause of mortality, as a critical test case, the thesis addresses, for the first time at large scale, challenges along the entire data and modelling pipeline. These span from data acquisition and processing, which I address by including a new methodology for digitising paper-based records, to the integration of waveforms within multimodal pipelines and the final combination of multimodal inputs.

The results show that multimodal integration improves predictive accuracy only conditionally. Late fusion proves most robust under real-world data constraints, with the largest gains in high-uncertainty cases and within the first 24 hours of admission. Subgroup analysis reveals that per-group calibration helps only under specific distributional conditions and that fairness constraints selected on validation data do not reliably transfer to test data. These findings shift the focus from whether multimodal fusion helps to when, how and for whom it does so, offering a principled framework for evaluating multimodal prediction systems.

Authors: Krones, F

• DOI: 10.5287/ora-8j4rqx46r
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: medical imaging; clinical decision-making; multimodal machine learning; data fusion