CARVE改进了循环注意力机制，让模型更懂记忆内容，训练更快更省显存。
CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention
- 将门控机制限制在键轴，实现内存感知的更新决策。
- 在1.3B参数下，困惑度降低0.18%，推理准确率提升0.63个百分点。
- 支持高效并行计算，训练速度比基线快1.4%~19.3%，适合大模型部署。
循环增量模型维护一个固定大小的状态矩阵 S（d_v × d_k），压缩所有历史上下文。当前最优方法（GDN-2）通过逐元素的擦除/写入掩码控制状态更新，但存在两个缺陷：一是门控仅依赖输入词元，导致模型对记忆内容无感知；二是擦除门控在值轴上的耦合阻碍了高效的WY形式分块求解器——使得块内系统分裂为d_v个独立求解，吞吐量退化为串行递推成本。本文提出CARVE（Content-Aware Recurrent with Value Efficiency），通过将所有门控约束于键轴，使块内耦合与值索引解耦，恢复单次未修改的WY形式求解。在此框架下，门控条件基于每块边界状态读取一次的内容信号，并代数折叠进低秩投影中（利用结合律，U(Sq)=(US)q），实现内存感知门控且额外通信开销极小。初始化时内容投影为零，CARVE与基线比特完全一致；我们证明单块过时对门控扰动仅为O(1/sqrt(L))，实测在L=128以内偏差仅0.18%。在1.3B参数、100B FineWeb-Edu数据上，使用H100（三组种子）测试，CARVE在各指标上均优于基线：WikiText困惑度15.72 vs 15.90（混合模式15.41 vs 15.62），常识准确率平均提升0.63个百分点，达到当前最优的RULER与真实场景召回表现；训练速度在相同深度下快1.4%，在等质量深度下快19.3%，峰值显存增加13%。论文提供六项形式化保证。
原文摘要 · Abstract (English)
Recurrent delta-rule models keep a fixed-size state matrix S (d_v x d_k) that compresses all past context. The state of the art (GDN-2) gates this update with element-wise matrix erase/write masks. This is powerful but has two defects. First, both gates are computed from the incoming token alone, making the model memory-blind: it decides what to erase without seeing what it has stored. Second, value-axis coupling in the erase gate blocks the WY-form triangular chunk solver that drives efficient training -- the intra-chunk system splits into d_v independent solves, collapsing throughput to serial-recurrence cost. We introduce CARVE (Content-Aware Recurrent with Value Efficiency), which fixes both and, via a single-launch "megakernel" scheduling of the same WY-form math, trains faster than the matrix-gated baseline it replaces. The key idea is architectural: restricting all gating to the key axis makes the intra-chunk coupling independent of the value index, restoring one unmodified WY-form solve. Within this constraint, CARVE conditions both gates on a content signal read once per chunk from the chunk-boundary state and folded algebraically into each gate's low-rank projection (by associativity, U(Sq)=(US)q), giving memory-aware gating at negligible extra traffic. At init the content projections are zero, so CARVE is bit-identical to the baseline; we prove the one-chunk staleness perturbs gates by only O(1/sqrt(L)), matching a measured 0.18% deviation flat up to L=128. At 1.3B parameters / 100B FineWeb-Edu tokens on H100 (three seeds), CARVE improves every axis: WikiText perplexity 15.72 vs 15.90 (hybrid 15.41 vs 15.62), +0.63 pp average common-sense accuracy, and state-of-the-art RULER and real-world recall -- while training +1.4% faster at matched depth and +19.3% at iso-quality depth, at +13% peak memory. Backed by six formal guarantees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。