用ICL的内部计算方式改进SFT模型表现
IA2: Alignment with ICL Activations Improves Supervised Fine-Tuning
- 通过自蒸馏让SFT模型模仿ICL的激活模式
- 在12个基准上提升模型准确率和输出校准度
- 适合关注模型推理机制与性能优化的研究者
监督微调（SFT）通过训练权重使模型对查询产生期望响应，而上下文学习（ICL）则在推理时通过提示中的指令或示例动态调整模型。尽管ICL在数据稀缺场景下具有更强泛化能力且输出更校准，但需更多推理计算。本文提出：能否利用ICL的内部计算来提升SFT？我们发现ICL与SFT产生不同的激活模式，表明二者通过不同功能机制实现适应。受此启发，我们提出ICL激活对齐（IA2），一种自蒸馏方法，旨在让SFT模型复制ICL的激活模式，并激励其进行类似ICL的内部推理。在12个主流基准和两个模型家族上的实验表明，将IA2作为SFT前的预处理步骤，显著提升了模型输出的准确率与校准度。这一发现不仅实用，也为理解模型适应的内在机制提供了新视角。
原文摘要 · Abstract (English)
Supervised Fine-Tuning (SFT) is used to specialize model behavior by training weights to produce intended target responses for queries. In contrast, In-Context Learning (ICL) adapts models during inference with instructions or demonstrations in the prompt. ICL can offer better generalizability and more calibrated responses compared to SFT in data scarce settings, at the cost of more inference compute. In this work, we ask the question: Can ICL's internal computations be used to improve the qualities of SFT? We first show that ICL and SFT produce distinct activation patterns, indicating that the two methods achieve adaptation through different functional mechanisms. Motivated by this observation and to use ICL's rich functionality, we introduce ICL Activation Alignment (IA2), a self-distillation technique which aims to replicate ICL's activation patterns in SFT models and incentivizes ICL-like internal reasoning. Performing IA2 as a priming step before SFT significantly improves the accuracy and calibration of model outputs, as shown by our extensive empirical results on 12 popular benchmarks and two model families. This finding is not only practically useful, but also offers a conceptual window into the inner mechanics of model adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。