让大模型在不更新权重的情况下，通过优化示例记忆提升少样本学习效果。
Context Tuning for In-Context Optimization
- 用示例初始化可训练记忆，再用梯度优化改进。
- 在多个基准上超越传统提示和上下文学习方法。
- 适合需要快速适应但资源有限的场景。
我们提出上下文调优（Context Tuning），一种无需更新权重即可显著提升大语言模型少样本适应能力的简单有效方法。上下文学习（ICL）在单次前向传播中形成示例的记忆表征，但当示例不足时无法优化。基于提示的方法通过优化可训练提示或前缀实现轻量级适应，但其初始化独立于示例。相比之下，上下文调优利用模型固有的上下文学习能力，从示例中初始化可训练记忆表征，并通过梯度优化进行精炼。在CrossFit、UnifiedQA、MMLU、BIG-Bench Hard和ARC等基准上的广泛评估表明，上下文调优优于传统提示方法和上下文学习，且与测试时训练（Test-Time Training）性能相当，但训练效率显著更高。
原文摘要 · Abstract (English)
We introduce Context Tuning, a simple and effective method to significantly enhance few-shot adaptation of large language models (LLMs) without weight updates. In-Context Learning (ICL) forms a memory representation of the demonstrations in a single forward pass but cannot refine it when insufficient. Prompt-based methods offer lightweight adaptation by optimizing a trainable prompt or prefix but initialize it independently of the demonstrations. In contrast, Context Tuning leverages the model's inherent ICL ability to initialize a trainable memory representation from demonstrations, then refines it through gradient-based optimization. Extensive evaluations on benchmarks such as CrossFit, UnifiedQA, MMLU, BIG-Bench Hard, and ARC demonstrate that Context Tuning outperforms both ICL and traditional prompt-based adaptation methods while achieving competitive accuracy with Test-Time Training at significantly higher training efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。