揭示深度Transformer表达能力的理论边界，证明层数越深越强
Knee-Deep in C-RASP: A Transformer Depth Hierarchy
- 用C-RASP编程语言建模深度变换器，发现其表达能力与层数严格对应
- 证明更深的模型能解决更复杂的序列依赖任务，且预测了长度泛化所需层数
- 理论适用于带位置编码的模型，对模型设计有指导意义
研究发现，变换器的深度（层数）与其能力正相关，但具体获得哪些能力尚不明确。本文通过理论证明与实证研究回答该问题：首先，考虑在注意力外仅保留固定精度的变换器，发现其表达能力等价于编程语言C-RASP，且深度保持不变；其次，证明更深的C-RASP程序比浅层更具表达力，由此推出在该子类中更深的变换器也更强大。该结论同样适用于含位置编码（如RoPE、ALiBi）的变换器。理论基础是与C-RASP等价的带计数算子时序逻辑。最后，实验验证了该理论能准确预测无位置编码变换器在一类序列依赖任务上实现长度泛化的最小深度要求。
原文摘要 · Abstract (English)
It has been observed that transformers with greater depth (that is, more layers) have more capabilities, but can we establish formally which capabilities are gained? We answer this question with a theoretical proof followed by an empirical study. First, we consider transformers that round to fixed precision except inside attention. We show that this subclass of transformers is expressively equivalent to the programming language C-RASP and this equivalence preserves depth. Second, we prove that deeper C-RASP programs are more expressive than shallower C-RASP programs, implying that deeper transformers are more expressive than shallower transformers (within the subclass mentioned above). The same is also proven for transformers with positional encodings (like RoPE and ALiBi). These results are established by studying a temporal logic with counting operators equivalent to C-RASP. Finally, we provide empirical evidence that our theory predicts the depth required for transformers without positional encodings to length-generalize on a family of sequential dependency tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。