TiBiX: leveraging temporal information for bidirectional X-ray and report generation

Conference item

With the emergence of vision language models in the medical imaging domain, numerous studies have focused on two dominant research activities: (1) report generation from Chest X-rays (CXR), and (2) synthetic scan generation from text or reports. Despite some research incorporating multi-view CXRs into the generative process, prior patient scans and reports have been generally disregarded. This can inadvertently lead to the leaving out of important medical information, thus affecting generation quality. To address this, we propose TiBiX: Leveraging Temporal information for Bidirectional X-ray and Report Generation. Considering previous scans, our approach facilitates bidirectional generation, primarily addressing two challenging problems: (1) generating the current image from the previous image and current report and (2) generating the current report based on both the previous and current images. Moreover, we extract and release a curated temporal benchmark dataset derived from the MIMIC-CXR dataset, which focuses on temporal data. Our comprehensive experiments and ablation studies explore the merits of incorporating prior CXRs and achieve state-of-the-art (SOTA) results on the report generation task. Furthermore, we attain on-par performance with SOTA image generation efforts, thus serving as a new baseline in longitudinal bidirectional CXR-to-report generation. The code is available at https://github.com/BioMedIA-MBZUAI/TiBiX.

Authors: Sanjeev, S; Maani, FA; Abzhanov, A; Papineni, VR; Almakky, I

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Springer Nature
• Host title: 4th MICCAI Workshop, DGM4MICCAI 2024, Held in Conjunction with MICCAI 2024, Marrakesh, Morocco, October 10, 2024, Proceedings
• Pages: 169-179
• Series: Lecture Notes in Computer Science
• Series number: 15224
• Place of publication: Cham, Switzerland
• Publication date: 2024-10-09
• Acceptance date: 2024-07-15
• Event title: DGM4MICCAI 2024
• Event website: https://caption-workshop.github.io/
• Event start date: 2024-10-06
• Event end date: 2024-10-06
• DOI: 10.1007/978-3-031-72744-3_17
• EISBN: 9783031727443
• ISBN: 9783031727436
• Language: English
• Keywords: longitudinal data; report generation; chest X-ray generation; multimodal data; bidirectional generation