提出Q-Delta，让查询参与状态更新，提升长文本建模效率与准确率。
Q-Delta: Beyond Key-Value Associative State Evolution
- 设计查询感知的增量规则，将查询与键联合用于状态更新
- 在语言建模和长上下文检索任务中均优于主流基线模型
- 支持硬件高效并行计算，适合部署于实际系统
线性注意力将序列建模重构为递归状态演化，实现线性时间推理。现有方法将查询仅用于读出操作，使其与状态演化解耦。本文揭示查询条件下的状态读出可生成对累积记忆的结构化值预测，补充键式检索。基于此，提出Q-Delta：一种查询感知的增量规则，将混合键-查询预测误差融入状态演化，实现协同修正动态，同时保持增量规则的高效性。建立了该动态系统的稳定性保证，并推导出硬件友好的分块并行形式，采用自定义Triton实现。实证结果表明，优化稳定，吞吐量具有竞争力，在语言建模与长上下文检索任务中持续超越强基线。
原文摘要 · Abstract (English)
Linear attention reformulates sequence modeling as recurrent state evolution, enabling efficient linear-time inference. Under the key-value associative paradigm, existing approaches restrict the role of the query to the readout operation, decoupling it from state evolution. We show that query-conditioned state readout induces a structured value prediction over accumulated memory that complements key-based retrieval. Based on this insight, we propose Q-Delta, a query-aware delta rule that integrates mixed key-query prediction errors into state evolution, enabling jointly corrective dynamics while preserving delta-rule efficiency. We establish stability guarantees for the resulting dynamics and derive a hardware-efficient chunkwise-parallel formulation with a custom Triton implementation. Empirical results demonstrate stable optimization, competitive throughput, and consistent improvements over strong baselines on language modeling and long-context retrieval tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。