在隐含模式任务中，上下文学习比微调更有效。
Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning
- 用上下文学习让模型快速捕捉隐含规律
- 微调用数千倍样本仍远不如上下文学习准确
- 适合关注模型内在机制与高效推理的研究者
微调和上下文学习（ICL）是赋予大语言模型特定任务知识的两种主流方法。普遍认为，只要有足够训练样本，微调能超越ICL，因为它可调整模型内部参数。然而本文提出反直觉发现：对于具有隐含模式的任务，ICL对模式的捕捉能力显著优于微调。研究构建了多个包含隐含模式的数据集，如通过奇偶性决定答案的序列、计算中可约项的识别等。在0.5B至7B参数的模型上，对比微调与ICL的表现。结果显示，采用ICL的模型能迅速掌握深层规律，准确率大幅提升；而微调虽使用数千倍于ICL的训练样本，改进却十分有限。此外，从机制可解释性视角提出了电路迁移理论，以解释ICL为何更具优势。
原文摘要 · Abstract (English)
Fine-tuning and in-context learning (ICL) are two prevalent methods in imbuing large language models with task-specific knowledge. It is commonly believed that fine-tuning can surpass ICL given sufficient training samples as it allows the model to adjust its internal parameters based on the data. However, this paper presents a counterintuitive finding: For tasks with implicit patterns, ICL captures these patterns significantly better than fine-tuning. We developed several datasets featuring implicit patterns, such as sequences determining answers through parity or identifying reducible terms in calculations. We then evaluated the models' understanding of these patterns under both fine-tuning and ICL across models ranging from 0.5B to 7B parameters. The results indicate that models employing ICL can quickly grasp deep patterns and significantly improve accuracy. In contrast, fine-tuning, despite utilizing thousands of times more training samples than ICL, achieved only limited improvements. We also proposed circuit shift theory from a mechanistic interpretability's view to explain why ICL wins.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。