提出理论框架，解释Transformer为何能或不能泛化到更长序列。
A Formal Framework for Understanding Length Generalization in Transformers
- 基于范数正则化构建理想推理下的可识别函数分析框架。
- 证明了在长输入下，一类丰富问题可实现长度泛化。
- 实验验证理论能准确预测算法与形式语言任务中的泛化成败。
Transformer面临的主要挑战之一是泛化到训练时未见的更长序列。尽管已有研究发现，Transformer在不同任务中可能成功或失败于长度泛化，但对其理论机制的理解仍有限。本文提出一个严谨的理论框架，用于分析具有可学习绝对位置编码的因果Transformer的长度泛化能力。特别地，在理想推理方案下，我们通过范数正则化刻画了从足够长输入中可识别的函数类别。这使我们能够证明一大类问题存在长度泛化的可能性。我们在一系列算法和形式语言任务上实验验证了该理论作为泛化成败预测器的有效性。该理论不仅解释了广泛存在的经验现象，还为可证明地预测Transformer的长度泛化能力开辟了道路。
原文摘要 · Abstract (English)
A major challenge for transformers is generalizing to sequences longer than those observed during training. While previous works have empirically shown that transformers can either succeed or fail at length generalization depending on the task, theoretical understanding of this phenomenon remains limited. In this work, we introduce a rigorous theoretical framework to analyze length generalization in causal transformers with learnable absolute positional encodings. In particular, we characterize those functions that are identifiable in the limit from sufficiently long inputs with absolute positional encodings under an idealized inference scheme using a norm-based regularizer. This enables us to prove the possibility of length generalization for a rich family of problems. We experimentally validate the theory as a predictor of success and failure of length generalization across a range of algorithmic and formal language tasks. Our theory not only explains a broad set of empirical observations but also opens the way to provably predicting length generalization capabilities in transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。