arXiv:2608.31067cs.LG2026-08

tiny Transformer可完美泛化任意长度的逻辑运算,仅280参数却能正确计算任意深度表达式。

Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers

论文配图:Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers
图 1 · 摘自论文原文
  • 将算法任务建模为电路嵌入Transformer,用深度位置编码实现逐层求值
  • 训练浅层问题后参数自动“锁定”,在任意长度上达到100%准确率
  • 适合研究符号计算、长度泛化和轻量级可解释模型的学者

神经网络在学习通用算法计算方面仍面临挑战,尤其在组合与长度泛化任务中表现不佳。本文提出一种可证明正确的Transformer参数化方法(布尔代数任务仅280个可学习参数),能够学习并评估任意深度或长度的问题。假设输入为完全括号化、语法正确的表达式。该方法将算法任务视为嵌入Transformer中的电路模型,可在单次前向传播中完成深度1的电路简化。为实现深度泛化,引入追踪每个门深度的位置编码,通过掩码硬注意力机制在每轮迭代中识别可求值子表达式,结合线性注意力实现每轮O(n)复杂度。配合自主终止条件,深度为d的问题仅需d轮迭代,总复杂度为O(n·d)。实验表明,在浅层问题(深度1和深度2)上训练后,参数会精确“锁定”,实现完全长度泛化。尽管本构造可严格证明对任意长度布尔表达式(一种通用符号计算)进行完美求值,其他实验也显示该Transformer变体在模块算术和ListOps等常见长度泛化基准上均实现100%准确率的完美泛化。

原文摘要 · Abstract (English)

Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and length generalization benchmarks. We present a provably correct, transformer parameterization (with only 280 learnable parameters for Boolean algebra tasks) capable of learning and evaluating problems of any depth or length. We assume inputs are fully parenthesized, well-formed expressions. Our approach conceptualizes algorithmic tasks as circuit models embedded in transformers, enabling depth-1 circuit reduction in a single forward pass. To achieve depth generalization, we introduce a positional encoding that tracks each gate's depth within the circuit, enabling the model to identify evaluable subexpressions at each iteration via masked hard attention, with $O(n)$ per-iteration complexity via linear attention. Combined with an autonomous halting criterion, the model terminates after $d$ iterations for problems of depth $d$, yielding $O(n \cdot d)$ total complexity. We show that training on shallow problem instances (depth 1 and depth 2) effectively recovers interpretable parameters that {\em snap} into place, resulting in exact length generalization. Though we establish that our construction provably evaluates Boolean expressions -- a universal symbolic computation -- of arbitrary length perfectly, in other experiments we also demonstrate that our transformer variant can learn and generalize perfectly (100% accuracy) on other common length generalization benchmarks, including modular arithmetic and ListOps.

符号计算长度泛化轻量模型Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。