提出关系优先的注意力机制，提升效率与性能
Ask Self, Ask Others: Relation Is All You Need
- 用显式自关系与交互关系组织信息流，替代传统注意力
- 在10M~100M参数模型上，全关系机制优于MHA，速度最高快4.41倍
- 适合追求高效低延迟推理的模型部署场景
注意力机制直接从成对得分中推导归一化信息流。本文提出一种新范式——关系（Relation），先将成对证据显式划分为自关系与交换关系，再由此推导信息流。该关系组织催生出全关系、闪速关系、线性关系、混合关系及基于键值的缓存机制。在约1000万、3000万和1亿参数的匹配解码器模型中，全关系在所有规模下均取得低于MHA的最终验证负对数似然。在固定上下文基准测试中，闪速关系比实际全关系实现快3.60至4.41倍。在匹配规模的生产负载下，其吞吐量可达PyTorch FlashAttention的76.4%至84.9%，同时执行全关系操作。混合关系使用75%线性关系层，仍保持优异语言建模质量。这些结果支持‘关系先行’的令牌混合观：先问自己，再问他人，最后让流动遵循关系。
原文摘要 · Abstract (English)
Attention directly derives normalized information flow from pairwise scores. We introduce Relation, an alternative token-mixing primitive that first organizes pairwise evidence into explicit Self and Exchange relations and derives information flow afterward. This relational organization gives rise to Full Relation, FlashRelation, Linear Relation, Hybrid Relation, and a KV-style Relation Cache. Across matched decoder-only models at approximately 10M, 30M, and 100M parameters, Full Relation achieves lower final validation NLL than MHA at all three scales. In a fixed-context reference benchmark, FlashRelation is 3.60-4.41x faster than the materialized Full Relation implementation. Across scale-matched production workloads, it reaches 76.4-84.9% of PyTorch FlashAttention throughput while executing the Full Relation operator. Hybrid Relation uses 75% Linear Relation layers and achieves strong language-modeling quality. These results support a relation-first view of token mixing: ask Self, ask Others, then let Flow follow Relation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。