研究提示词中早期例子如何干扰模型对后续任务的判断。
When Context Sticks: Studying Interference in In-Context Learning
- 用线性与二次函数任务测试提示中前后例子的干扰效应。
- 越多前置线性例子，二次预测误差越大；二次例子越多，纠错效果递减。
- 按目标函数顺序训练的模型恢复最快，随机训练最易受干扰。
本文研究提示词学习中的上下文粘滞现象，即提示中早期示例会干扰模型对后续任务的适应能力。通过在直线和抛物线函数上设计合成回归任务，我们考察了模型在顺序、混合和随机训练课程下，面对推理阶段突然任务切换的表现。通过系统测试一系列误导性线性示例后接恢复性二次示例的组合，量化了先前上下文对预测误差的影响程度及模型重校准速度。结果表明存在显著的持续干扰：前期线性示例越多，二次预测误差越高；尽管增加二次示例可降低误差，但改善效果随数量增加而递减。此外，训练课程显著影响抗干扰能力，按目标函数类别顺序训练的模型恢复最快，而随机训练则表现出最弱的鲁棒性。
原文摘要 · Abstract (English)
This paper investigates context stickiness in in-context learning (ICL), a phenomenon where earlier examples in a prompt interfere with a transformer's ability to adapt to later tasks. Using synthetic regression tasks over linear and quadratic functions, we examine how models trained under sequential, mixed, and random curricula handle abrupt task switches during inference. By sweeping over structured combinations of misleading linear examples followed by recovery quadratic examples, we quantify how prior context biases prediction error and how quickly models realign. Our results show strong evidence of persistent interference: more preceding linear examples reliably degrade quadratic predictions, while additional quadratic examples reduce error but with diminishing returns. We further find that training curricula significantly modulate resilience, with sequential training on the target function class yielding the fastest recovery, and surprisingly, random training producing the least robust behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。