arXiv:2506.09251cs.CLcs.AI2025-06NeurIPS被引 12

通过任务关联,让模型学会从短输入推断长输入。

Extrapolation by Association: Length Generalization Transfer in Transformers

  • 用相关任务辅助训练,提升模型对更长输入的泛化能力。
  • 在算术、字符串变换等任务中验证了长度泛化的可迁移性。
  • 发现注意力头复用是跨任务泛化的重要机制,适合研究模型泛化原理者阅读。

Transformer语言模型在自然语言领域展现出强大的泛化能力,但其泛化机制尚不清晰。本文从任务关联视角研究长度泛化——即从较短输入外推到更长输入的能力。我们发现,长度泛化可在相关任务间转移:在较长且相关的辅助任务上训练模型,能使其在其他目标任务中对未见过的更长输入实现泛化。我们在算术运算、字符串变换和迷宫导航等多种算法任务中验证了这一现象。结果表明,联合训练使模型能继承相似任务的泛化能力。此外,在预训练语言模型中也观察到类似转移效应,提示预训练赋予模型可复用的计算结构,有助于下游任务中的外推。最后,我们提供了初步的机制证据:长度泛化转移与任务间注意力头的重复使用相关。这些发现深化了对Transformer模型外分布输入泛化机制的理解,并揭示了归纳结构在任务间的组合复用特性。

原文摘要 · Abstract (English)

Transformer language models have demonstrated impressive generalization capabilities in natural language domains, yet we lack a fine-grained understanding of how such generalization arises. In this paper, we investigate length generalization--the ability to extrapolate from shorter to longer inputs--through the lens of \textit{task association}. We find that length generalization can be \textit{transferred} across related tasks. That is, training a model with a longer and related auxiliary task can lead it to generalize to unseen and longer inputs from some other target task. We demonstrate this length generalization transfer across diverse algorithmic tasks, including arithmetic operations, string transformations, and maze navigation. Our results show that transformer models can inherit generalization capabilities from similar tasks when trained jointly. Moreover, we observe similar transfer effects in pretrained language models, suggesting that pretraining equips models with reusable computational scaffolding that facilitates extrapolation in downstream settings. Finally, we provide initial mechanistic evidence that length generalization transfer correlates with the re-use of the same attention heads between the tasks. Together, our findings deepen our understanding of how transformers generalize to out-of-distribution inputs and highlight the compositional reuse of inductive structure across tasks.

长度泛化任务关联注意力机制Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。