提出一种内存高效的量子启发注意力机制,提升文本生成质量。
Q-Interference: Memory-Efficient Phase-Aware Quantum-Inspired Attention

- 给查询和键增加幅度与可学习相位,实现相位感知的交互
- 通过三角恒等式分解,避免大中间张量,内存降低60%以上
- 可直接嵌入GPT模型,适合追求高效生成的开发者
GPT中的注意力通过点积相似度衡量词元兼容性,简单有效但未显式建模强特征间的增强或抑制关系。本文提出Q-Interference,一种全经典的量子启发注意力机制,在每个查询和键特征上增加幅度和可学习相位。注意力分数具有相位感知特性:同相位贡献相加,反相位则相互抵消。尽管这种机制更丰富,但原始实现需大规模词元对特征交互张量,内存开销大。为此,我们提出精确的三角恒等式分解方法,仅用两次标准矩阵乘法即可计算相同结果,避免中间张量的显式构造。Q-Interference可直接集成至GPT结构中,保持原有架构与下一个词预测目标不变。在公开基准数据集上的实验表明,该重构方法在控制的GPT式设置下训练稳定,并相比朴素相位感知注意力实现一致的内存优势。结果验证了本文核心贡献:一种精确的内存高效重构,使相位感知干扰注意力在标准GPT流水线中具备实用性。
原文摘要 · Abstract (English)
GPT attention measures token compatibility through dot-product similarity. This mechanism is simple, effective, and memory-efficient. But it does not explicitly model whether strong token features should reinforce or suppress one another. We introduce Q-Interference, a fully classical quantum-inspired attention mechanism for autoregressive language modeling that augments each query and key feature with an amplitude and a learned phase. The resulting attention score is phase-aware which aligned phases contribute constructively while conflicting phases contribute destructively. Although Q-Interference yields a richer interaction rule than similarity alone, a naive implementation of Q-Interference requires a large token-pair-feature interaction tensor, making it memory-intensive and often impractical. To address this limitation, we propose an exact trigonometric factorization that computes the same score using two standard matrix multiplications avoiding materialization of the large intermediate tensor. Q-Interference fits directly into a Transformer block in GPT and leaves the remainder of the model architecture and next-token prediction objective unchanged. Experiments on public benchmark datasets and baseline models show that the proposed reformulation trains stably in a controlled GPT-style setting and provides a consistent memory advantage over naive phase-aware interference attention. These results support the specific contribution of this work: an exact memory-efficient reformulation that makes phase-aware interference attention practical within a standard GPT pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。