提出相位一致的Transformer,提升复数域注意力的稳定性与泛化能力。
Complex-Valued Phase-Coherent Transformer
- 用平滑门控替代软最大值,保持查询键相似性的相位信息
- 在多个任务上超越标准Transformer和同类复数模型,长程记忆表现突出
- 适合需要精确相位保留的复杂信号建模场景,如语音、图像等
复数域Transformer大多沿用实数域的softmax注意力机制,但行归一化的令牌竞争并不必然与相位保持计算对齐。本文提出相位一致Transformer(PCT),通过在L2归一化的复数查询-键相似度上应用实值、元素无关且平滑的门控机制,将令牌竞争改为非竞争注意力,以实现跨层相位信息保持。在涵盖长程记忆、分层长程推理、位置检索、基于相位的记忆与叠加等中等规模基准测试中,PCT展现出强大的任务泛化能力。在参数公平比较下,其持续优于标准softmax Transformer及其直接复数版本。即使在传统上对复数神经网络困难的任务(如NIAH和LRA-Text)上,PCT仍可与最强实数基线Multiscreen竞争。引入违反PCT设计条件的门控实验表明:保持负向对齐相位成分的平滑门控仍具强性能,而删除这些成分或输出过大的门控则导致长程检索崩溃和性能明显下降。PCT在测试深度范围内无深度相关精度衰减。结果支持将多层相位一致结构引入注意力,作为复数域Transformer实现泛化的重要设计原则。
原文摘要 · Abstract (English)
Complex-valued Transformers have largely inherited softmax attention from real-valued architectures. However, row-normalised token competition is not necessarily aligned with phase-preserving computation. In this paper, we introduce the Phase-Coherent Transformer (PCT), which applies a real-valued, element-independent, smooth gate to L2-normalised complex query-key similarities. PCT replaces token competition with token-non-competing attention and is designed to preserve phase information across layers. Across mid-scale benchmarks spanning long-range memory, hierarchical long-range reasoning, positional retrieval, phase-based memory and superposition, and image classification, PCT shows strong generalisation across task categories. Under parameter-fair comparison, PCT consistently outperforms both the standard softmax Transformer and its direct complex-valued counterpart. Moreover, even on tasks traditionally considered difficult for complex-valued neural networks, such as NIAH and LRA-Text, PCT remains competitive with Multiscreen, the strongest real-valued NN baseline in our comparison. Experiments introducing gates that deliberately violate the PCT conditions show that the design is not incidental: smooth gates that preserve negatively aligned phase components remain strong, whereas gates that delete such components collapse on long-range retrieval, and gates whose outputs become excessively large suffer clear performance degradation. PCT also shows no depth-related accuracy collapse across the tested depth range. These results support introducing multi-layer phase-coherent structure into attention as a promising design principle for achieving generalisation in complex-valued Transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。