混合架构用常数级思维链解决复杂检索任务
Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders
- 融合门控注意力与递归DeltaNet的混合解码器
- 在固定精度下仅需常数级思维链步骤完成任务
- 适合研究模型表达能力与高效推理的学者
我们研究了混合循环-注意力解码器的表达能力，这类架构被用于Qwen3-Next等开源语言模型。模型结合了门控注意力头与递归门控DeltaNet头。是否存在形式上的优势？我们定义了受奇偶性约束的检索任务，并证明在常数精度假设下，类Qwen的混合结构可使用常数规模的临时存储（即$O(1)$思维链步骤）解决该任务。相比之下，纯门控DeltaNet无法实现类似解法，而纯门控注意力至少需要多项式规模临时存储。
原文摘要 · Abstract (English)
We investigate the expressive power of hybrid recurrent-attention decoders, a class of architectures used in recent open-source language models such as Qwen3-Next and its successors. These models combine Gated Attention heads with recurrent Gated DeltaNet heads. Is there a formal advantage, in terms of model expressivity or efficiency, to such a hybrid architecture? We show that there is. We define parity-conditioned retrieval task and show that under constant-precision assumption, a Qwen-style hybrid of Gated DeltaNet and Gated Attention solves this task with a constant scratchpad, or equivalently $O(1)$ chain-of-thought steps. In contrast, no similar solution exists for pure Gated DeltaNet models, while pure Gated Attention requires at least a polynomial scratchpad.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。