Retrieval-augmented AI assistants for healthcare: System design and evaluation

Thesis

This dissertation explores the application of Retrieval-Augmented Generation (RAG) in AI models to address the challenges posed by fragmented personal health data. Contemporary healthcare systems often struggle with scattered patient information, hindering efficient and informed decision-making. This work investigates whether a RAG-enhanced AI assistant can reliably answer natural-language queries over personal health records, focusing on accuracy, transparency, and auditability using only synthetic data for privacy.

The primary contribution is a modular, local-first prototype system designed to ingest, chunk, embed, and retrieve information from PDF health records to ground AI-generated answers. A comprehensive evaluation was conducted, systematically varying RAG parameters like chunk size, retrieval depth, and embedding models across different query types (factual, multi-document, contradiction). This dissertation presents findings on the trade-offs between retrieval precision, answer correctness, latency, and cost. A detailed failure mode analysis links design choices to risks like fact fabrication and missed context, revealing that AI model reasoning, rather than information retrieval, is often the main bottleneck for complex queries. Ethical, legal, and social implications are examined, proposing safeguards such as verifiable citations and calibrated uncertainty. The work concludes by reflecting on the design, evaluation, and potential for scaling such systems responsibly.

Authors: Meleka, M

• DOI: 10.5287/ora-vgvoyqkp9
• Type of award: MSc
• Level of award: Masters
• Awarding institution: University of Oxford
• Language: English
• Keywords: AI; retrieval-augmented generation; healthcare; RAG; artificial intelligence
• Subjects: Retrieval-Augmented Generation; Artificial intelligence