提出可复用的隐式上下文学习方法,一次训练全局通用。
Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention
- 通过注意力日志捕捉通用上下文模式,用可学习路由器调节注意力。
- 在12个数据集上超越现有方法,跨领域任务表现更稳定。
- 适合追求零样本高效推理与模型泛化能力的研究者。
隐式上下文学习(ICL)作为新兴范式,旨在模拟大语言模型(LLMs)中上下文学习行为,实现少样本性能而无需额外训练成本。然而,现有方法多依赖从标注示例或特定任务对齐中生成的偏移向量注入残差流,未能充分利用ICL的内在结构机制,且泛化能力有限。为此,我们提出上下文路由(ICR),一种新型隐式ICL方法,能在注意力日志层面捕捉并利用可复用的通用ICL模式。该方法提取上下文学习过程中自然出现的结构性方向,并采用可学习的输入条件路由器动态调制注意力日志,实现只需一次训练即可广泛复用的框架。我们在12个涵盖不同领域的实际数据集及多个LLM上评估ICR,结果表明其持续优于需任务特异性检索或训练的现有方法,且在未见领域任务中仍保持稳健表现。这些发现推动了ICL实际应用价值的边界。代码已公开于https://github.com/Lijiaqian1/In-Context-Routing.git。
原文摘要 · Abstract (English)
Implicit in-context learning (ICL) has newly emerged as a promising paradigm that simulates ICL behaviors in the representation space of large language models (LLMs), aiming to attain few-shot performance at zero-shot cost. However, existing approaches largely rely on injecting shift vectors into residual flows, which are typically constructed from labeled demonstrations or task-specific alignment. Such designs fall short of utilizing the structural mechanisms underlying ICL and suffer from limited generalizability. To address this, we propose In-Context Routing (ICR), a novel implicit ICL method that captures and utilizes generalizable ICL patterns at the attention logits level. It extracts reusable structural directions that emerge during ICL and employs a learnable input-conditioned router to modulate attention logits accordingly, enabling an efficient train-once-and-reuse framework. We evaluate ICR on 12 real-world datasets spanning diverse domains and multiple LLMs. The results show that ICR consistently outperforms existing implicit ICL methods that require task-specific retrieval or training, while demonstrating robust generalization to out-of-domain tasks where they struggle. These findings position ICR to push the boundary of the practical value of ICL. The code is available at https://github.com/Lijiaqian1/In-Context-Routing.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。