将大模型推理视为优化过程，实现高效少样本适配。
Optimization-Inspired Few-Shot Adaptation for Large Language Models
- 把模型前向传播看作带预条件的梯度下降，设计无额外参数的适配机制。
- 在多个少样本任务上性能超越现有方法，收敛更快且更稳定。
- 适合资源受限场景下的模型快速部署，尤其适合小样本应用。
大型语言模型在实际应用中表现卓越，但通过微调适应新任务通常需要大量数据和计算资源，这在少样本场景下不切实际。现有方法如上下文学习（in-context learning）增加推理开销且提升有限，而参数高效微调（PEFT）则容易过拟合少量示例。本文将大模型的前向传播重新诠释为一系列预条件梯度下降步骤，用于优化内部表示。基于此，提出优化启发的少样本适配方法（OFA），引入无需额外可训练参数的预条件器参数化，并设计基于收敛边界的学习目标，以提升优化效率并引导优化路径趋向平坦局部极小值。实验表明，该方法克服了基于ICL与PEFT方法的缺陷，在多种少样本适配任务上表现更优。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable performance in real-world applications. However, adapting LLMs to novel tasks via fine-tuning often requires substantial training data and computational resources that are impractical in few-shot scenarios. Existing approaches, such as in-context learning and Parameter-Efficient Fine-Tuning (PEFT), face key limitations: in-context learning introduces additional inference computational overhead with limited performance gains, while PEFT models are prone to overfitting on the few demonstration examples. In this work, we reinterpret the forward pass of LLMs as an optimization process, a sequence of preconditioned gradient descent steps refining internal representations. Based on this connection, we propose Optimization-Inspired Few-Shot Adaptation (OFA), integrating a parameterization that learns preconditioners without introducing additional trainable parameters, and an objective that improves optimization efficiency by learning preconditioners based on a convergence bound, while simultaneously steering the optimization path toward the flat local minimum. Our method overcomes both issues of ICL-based and PEFT-based methods, and demonstrates superior performance over the existing methods on a variety of few-shot adaptation tasks in experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。