Do as I do (safely): mitigating task-specific fine-tuning risks in large language models

Conference item : Poster

Recent research shows that fine-tuning on benign instruction-following data can inadvertently undo the safety alignment process and increase a model's propensity to comply with harmful queries. While instruction-following fine-tuning is important, task-specific fine-tuning-where models are trained on datasets with clear ground truth answers (e.g., multiple choice questions)-can enhance model performance on specialized downstream tasks. Understanding and mitigating safety risks in the task-specific setting remains distinct from the instruction-following context due to structural differences in the data. Our work demonstrates how malicious actors can subtly manipulate the structure of almost any task-specific dataset to foster significantly more dangerous model behaviors, while maintaining an appearance of innocuity and reasonable downstream task performance. To address this issue, we propose a novel mitigation strategy that mixes in safety data which mimics the task format and prompting style of the user data, showing this is significantly more effective and efficient than existing baselines at re-establishing safety alignment while maintaining similar task performance.

Authors: Eiras, F; Petrov, A; Torr, P; Kumar, MP; Bibi, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publication date: 2025-01-22
• Acceptance date: 2025-01-22
• Event title: Thirteenth International Conference on Learning Representations (ICLR 2025)
• Event series: International Conference on Learning Representations
• Event location: Singapore
• Event website: https://iclr.cc/Conferences/2025
• Event start date: 2025-04-24
• Event end date: 2025-04-28
• Language: English
• Keywords: large language models; safety; fine-tuning
• Subtype: Poster