揭示任意顺序生成中结构与语义注意力的内在矛盾，解释为何双流注意力更优。
Why Any-Order Autoregressive Models Need Two-Stream Attention: A Structural-Semantic Tradeoff
- 提出结构-语义权衡机制，解释单流注意力在任意顺序生成中的瓶颈。
- 在短序列上，解耦旋转位置编码性能接近双流注意力，长序列则显著下降。
- 适合研究高效自回归模型、注意力机制设计的学者参考。
任意顺序自回归模型（AO-ARMs）通过原生键值缓存实现高效掩码扩散，但高性能需依赖双流注意力，传统解释为分离内容与位置。本文指出，双流注意力可能解决更深层问题：生成过程中隐藏表示需同时关注语义相关词以预测，又需关注结构上近期词以摘要。这两者在单流中竞争注意力容量，但在双流中可分工。为剥离位置-内容分离的影响，本文提出解耦旋转位置编码（Decoupled RoPE），提供目标位置信息而不泄露内容。该方法在短序列（语义与结构邻近）表现良好，但随序列增长而退化。结果表明，双流注意力成功的关键并非单纯分离位置与内容，而是规避了任意顺序生成固有的结构-语义冲突。
原文摘要 · Abstract (English)
Any-order autoregressive models (AO-ARMs) offer a promising path toward efficient masked diffusion by enabling native key-value caching, but competitive performance has so far required two-stream attention, typically motivated as a means of decoupling token content from position. In this work, we argue that two-stream attention may be serving a more subtle role. We identify a structural-semantic tradeoff in any-order generation: the hidden representation at each step must simultaneously attend to semantically informative tokens for prediction and structurally recent tokens for summarization, objectives that compete for attention capacity in a single stream but can specialize across two streams. To isolate this tradeoff from position-content separation, we propose Decoupled RoPE, a modification to rotary position embeddings that provides target position information without revealing target content. Decoupled RoPE performs competitively at short sequence lengths--where semantic and structural proximity coincide--but degrades as sequence length increases and the two orderings diverge. These results suggest that the success of two-stream attention stems not merely from separating position from content, but from circumventing the deeper structural-semantic tradeoff inherent to any-order generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。