Optimization inspired few-shot adaptation for Large Language Models

Conference item

Large Language Models (LLMs) have demonstrated remarkable performance in real-world applications. However, adapting LLMs to novel tasks via finetuning often requires substantial training data and computational resources that are impractical in few-shot scenarios. Existing approaches, such as In-context learning and Parameter-Efficient Fine-Tuning (PEFT), face key limitations: Incontext learning introduces additional inference computational overhead with limited performance gains, while PEFT models are prone to overfitting on the few demonstration examples. In this work, we reinterpret the forward pass of LLMs as an optimization process, a sequence of preconditioned gradient descent steps refining internal representations. Based on this connection, we propose Optimization-Inspired Few-Shot Adaptation (OFA), integrating a parameterization that learns preconditioners without introducing additional trainable parameters, and an objective that improves optimization efficiency by learning preconditioners based on a convergence bound, while simultaneously steering the optimization path toward the flat local minimum. Our method overcomes both issues of ICL-based and PEFT-based methods, and demonstrates superior performance over the existing methods on a variety of few-shot adaptation tasks in experiments.

Authors: Gao, B; Wang, X; Yang, Y; Clifton, D

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: NeurIPS
• Volume: 38
• Pages: 25782-25814
• Publication date: 2026-05-01
• Acceptance date: 2025-09-19
• Event title: 39th Annual Conference on Neural Information Processing Systems (NeurIPS 2025)
• Event location: San Diego, CA, USA
• Event website: https://neurips.cc/Conferences/2025
• Event start date: 2025-12-02
• Event end date: 2025-12-07
• Language: English