揭示Transformer在哪些规则语言上能泛化长度,给出可高效判断的方法。
Algebraic Decomposition Theory for Transformer Length Generalization
- 提出C-RASP形式化框架,刻画Transformer长度泛化的语言范围。
- 发现经典分解理论不适用,需引入整数加法群的新型代数结构。
- 给出多项式时间判定算法,实验验证优于现有分类方法。
基于Transformer的语言模型有时能在训练时未见的更长序列上实现泛化,但我们尚缺乏对哪些任务具备长度泛化能力的精确刻画。甚至对于最基本的规则语言类别,也未明确其是否能被变压器泛化。本文首次完整刻画了哪些规则语言具备长度泛化特性,并提供了在语言语法幺半群规模上多项式时间运行的决策算法。该成果依赖于对C-RASP(一种近期建立的形式化框架)中可被变压器长度泛化的语言类别的有效刻画。这一刻画极具挑战性:经典有限半群的Krohn-Rhodes分解理论不足以支撑,因基本构建块(如翻转触发器与简单群)无法在C-RASP中表达;而C-RASP的基本构建块(无界计数)也无法由Krohn-Rhodes中的有限半群表达。因此,长度泛化受制于一个经典分解理论无法捕捉的代数性质。本文将经典分解理论从有限半群推广至整数上的无限加法群,从而以整数的迭代缠绕积刻画了C-RASP,并推导出针对规则语言成员关系的可证明多项式时间决策算法。在广泛的规则语言测试集上的实验表明,该理论比现有分类更准确地捕捉了Transformer的长度泛化行为。
原文摘要 · Abstract (English)
Transformer-based language models are known to sometimes generalize to sequences longer than seen during training, but we lack a precise characterization of which tasks admit length generalization. It is not even known which regular languages transformers length-generalize on -- and this is a foundational class of languages. Our contributions are to establish the first complete characterization of which regular languages transformers length-generalize on and provide a decision algorithm running in polynomial time in the size of the language's syntactic monoid. These results rely on an effective characterization of the regular languages in C-RASP, a recently-established formalism that expresses which languages transformers length-generalize on. This characterization is challenging because classical tools like Krohn-Rhodes decomposition theory for finite semigroups are insufficient for C-RASP. Firstly, the basic building blocks of Krohn-Rhodes theory -- flip-flop and simple groups -- are not expressible in C-RASP. Secondly, the basic building block of C-RASP (unbounded counting) is not expressible by the finite semigroups of Krohn-Rhodes theory. Thus, length generalization on regular languages is controlled by an algebraic property that is invisible to classical finite decomposition theory. We generalize classical decomposition theory from finite semigroups to the infinite additive group on the integers, allowing us to characterize C-RASP in terms of iterated wreath products of the integers and derive a provable polynomial-time decision algorithm for regular language membership. Experiments across a broad test suite of regular languages confirm that our theory captures transformers' length-generalization behavior more accurately than existing classifications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。