arXiv:2603.05829cs.LGcs.CL2026-03

用大量示例提示让大模型推理时自适应,但效果受策略影响大。

Test-Time Adaptation via Many-Shot Prompting: Benefits, Limits, and Pitfalls

  • 通过输入端注入多示例实现推理时自适应,不修改模型参数。
  • 结构化任务中性能提升明显,开放生成任务效果有限。
  • 示例选择和顺序影响巨大,需谨慎设计提示策略。

测试时自适应使大型语言模型在推理阶段无需更新参数即可调整行为。一种常见方法是多示例提示(many-shot prompting),即在输入中注入大量上下文学习(ICL)示例作为输入空间的测试时更新。尽管增加示例数量可提升性能,但该机制的可靠性与局限性,特别是对开源模型的影响,仍不明确。本文对多种任务和模型架构下的多示例提示进行实证研究,分析性能随更新强度、示例顺序及选择策略的变化。进一步考察动态与强化型ICL等替代策略,以控制注入信息并约束模型行为。结果表明,多示例提示在演示提供高信息增益的结构化任务中有效,但在开放生成任务中对选择策略极为敏感,且常收益有限。总体上,我们刻画了基于提示的测试时自适应的实际边界,并明确了输入空间更新何时有益、何时有害。

原文摘要 · Abstract (English)

Test-time adaptation enables large language models (LLMs) to modify their behavior at inference without updating model parameters. A common approach is many-shot prompting, where large numbers of in-context learning (ICL) examples are injected as an input-space test-time update. Although performance can improve as more demonstrations are added, the reliability and limits of this update mechanism remain poorly understood, particularly for open-source models. We present an empirical study of many-shot prompting across tasks and model backbones, analyzing how performance varies with update magnitude, example ordering, and selection policy. We further study Dynamic and Reinforced ICL as alternative test-time update strategies that control which information is injected and how it constrains model behavior. We find that many-shot prompting is effective for structured tasks where demonstrations provide high information gain, but is highly sensitive to selection strategy and often shows limited benefits for open-ended generation tasks. Overall, we characterize the practical limits of prompt-based test-time adaptation and outline when input-space updates are beneficial versus harmful.

测试时自适应提示工程LLM推理ICL

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。