用可学习门控机制实现高效循环注意力，让小模型逼近大模型性能。
CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability
- 共享核心块重复使用，仅预计算一次键值，通过隐式注意力跨层传递上下文。
- 在1024维下参数对齐时性能落后基准1-2%，有效参数对齐下差10%。
- 循环门控稳定性强，谱半径始终控制在0.79-0.83窄区间内，适合资源受限场景。
我们提出CART（Context-Anchored Recurrent Transformer），一种参数高效的语言模型，其通过在深度方向上重复使用单一共享核心块共R次。与以往循环变压器不同，CART仅在多层前导阶段计算一次键值（K、V），并让循环核心通过多头隐式注意力与这些冻结张量交互。一个可学习的线性时不变（LTI）门控机制确保递归稳定性：所有36个完整训练配置的谱半径均收敛于[0.79, 0.83]的狭窄范围。我们在单个消费级GPU上分两阶段评估：先进行64组配置在3,000步的筛选，再对36组配置（P=6，R∈{6,8,10}，三个随机种子）训练至30,500步（约10亿词元）。在宽度d∈{256,512,768,1024}下，前导深度P对性能影响大于循环次数R；且第一阶段排名随训练推进反转（如R=6在d≥512时成为最优）。在参数对齐测试中（d=1024），CART未超越参数匹配的密集基线，存储参数对齐下落后1-2%，有效参数对齐下落后约10%。诊断消融实验表明，有效参数差距中约5%来自权重共享，另约5%源于前导/锚点/核心/尾部结构不一致；而循环核心机制（超连接、LTI门控、循环索引嵌入）各自均为冗余。变量R推理在训练最优R两侧均出现性能下降，表明该方法不支持测试时动态扩展深度。
原文摘要 · Abstract (English)
We present CART (Context-Anchored Recurrent Transformer), a parameter-efficient language model that reuses a single shared core block R times across depth. Unlike prior looped transformers that recompute key-value tensors at every iteration, CART computes K and V once from a multi-layer prelude and has the recurrent core cross-attend to those frozen tensors via multi-head latent attention. A learned Linear Time-Invariant (LTI) gate keeps the recurrence stable: its spectral radius settles in a narrow band (rho in [0.79, 0.83]) across all 36 fully-trained configurations. We evaluate CART on single consumer GPUs in two stages: a 64-configuration screen at 3,000 steps, then 36 configurations (P=6, R in {6,8,10}, three seeds) trained for 30,500 steps (~1B tokens). Two patterns hold across widths d in {256,512,768,1024}: prelude depth P dominates loop count R, and the Stage-1 ranking of R reverses at full training (R=6 becomes best at d>=512). At the binding d=1024 parameter-parity test, CART does not beat a parameter-matched dense baseline, losing by 1-2% at stored-parameter parity and by ~10% at effective-parameter parity. Diagnostic ablations split the effective-parameter gap into ~5% from weight sharing and a residual ~5% from the heterogeneous prelude/anchor/core/coda framing; the recurrent-core machinery (hyper-connections, LTI gate, loop-index embedding) is individually vestigial. Variable-R inference degrades on both sides of the trained R, a negative result for test-time depth scaling under this recipe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。