Transformer在加法乘法中实现双长度泛化，首次突破极限。
Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count
- 用专用记事本让模型每步聚焦固定数量词元
- 多层级位置耦合提升注意力定位精度
- 理论证明1层模型可处理指数级长度与数量
Transformers常因长度泛化能力差而失效，即无法处理训练时未见的长序列。算术任务是研究长度泛化的常用场景，但多操作数加法（需同时泛化操作数数量与长度）和乘法（需泛化操作数长度）尤为困难。本文首次实现算术Transformer在两类任务上约2-3倍的长度泛化能力。通过设计任务特异性记事本机制，使模型在每步预测时仅关注固定数量词元；并引入多层级位置耦合（Position Coupling），帮助Transformer准确定位应关注的位置。理论上，使用该方法的1层Transformer可解决多操作数加法问题，其操作数长度与数量上限可达嵌入维度的指数级。
原文摘要 · Abstract (English)
Transformers often struggle with length generalization, meaning they fail to generalize to sequences longer than those encountered during training. While arithmetic tasks are commonly used to study length generalization, certain tasks are considered notoriously difficult, e.g., multi-operand addition (requiring generalization over both the number of operands and their lengths) and multiplication (requiring generalization over both operand lengths). In this work, we achieve approximately 2-3x length generalization on both tasks, which is the first such achievement in arithmetic Transformers. We design task-specific scratchpads enabling the model to focus on a fixed number of tokens per each next-token prediction step, and apply multi-level versions of \Position Coupling (Cho et al., 2024; McLeish et al., 2024) to let Transformers know the right position to attend to. On the theory side, we prove that a 1-layer Transformer using our method can solve multi-operand addition, up to operand length and operand count that are exponential in embedding dimension.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。