ATMA通过极化注意力与门控增量记忆，实现长文本建模的高效精准。
ATMA: Long-Context Language Modeling via Polar Attention and Gated-Delta Compression Memory
- 采用极化注意力分离方向与强度通道，结合门控增量记忆提升长序列处理能力。
- 在256K上下文中保持34.4%目标词准确率，检索准确率达18.0%（合成数据）。
- 适合追求长文本建模性能、关注模型稳定性与可复现性的研究者。
语言模型的长度外推面临多个相互竞争的目标：信息检索保真度、长文档似然性、短上下文质量与推理成本。我们提出ATMA，一个378M参数的混合架构，结合极化注意力与门控-增量递归记忆，并将这些目标视为帕累托问题而非追求通用架构优势。极化注意力将归一化方向通道与有界参与率幅度通道分离。通过完整120单元、10亿标记的因子实验筛选出最优配置，随后在2K长度上训练匹配的NoPE、RoPE与极化变体，共98.16亿标记，评估延伸至256K。在所有20个匹配单元中，记忆机制使极化注意力在64K检索任务中平均提升47.8分；对NoPE影响小且不一致。在256K时，极化注意力保留34.4%教师强制下的目标词准确率和9.0%五词精确匹配准确率；合成上下文中精确检索达18.0%，但细粒度PDF上下文中为0.0%。极化注意力将平均固定目标比特/字节退化控制在1.26倍以内，在8个短上下文任务上仅增加1.9分成本。Raven基线在长文本任务中优于BABILong，且解码状态与长度无关，代表了前沿上的另一点。最后，事后检查显示，相似的2K验证曲线可能隐藏6.70纳特的256K差异，由于未种子配对且设备分布非随机，这被解释为基础设施迁移引发的检查点变异性，而非因果硬件效应。代码：https://github.com/kreasof-ai/atma
原文摘要 · Abstract (English)
Length extrapolation in language models involves competing objectives: retrieval fidelity, long-document likelihood, short-context quality, and inference cost. We present ATMA, a 378M-parameter hybrid recipe that combines Polar Attention with gated-delta recurrent memory, and study these objectives as a Pareto problem rather than claiming general architectural dominance. Polar Attention separates a normalized direction channel from a bounded participation-ratio magnitude channel. We select the recipe with a complete 120-cell, 1B-token factorial sweep, then train matched NoPE, RoPE, and Polar variants for 9.816B tokens at length 2K and evaluate them through 256K. Across the factorial, memory improves Polar's 64K retrieval score in all 20 matched cells (mean +47.8 points), whereas its effect on NoPE is small and inconsistent. At 256K, Polar retains 34.4% teacher-forced target-token accuracy and 9.0% exact five-token accuracy; exact retrieval is 18.0% on synthetic contexts but 0.0% on FinePDFs contexts. Polar also limits mean fixed-target bits-per-byte degradation to 1.26 times, at a 1.9-point mean cost on eight short-context tasks. Raven baselines lead BABILong and have length-independent decode state, illustrating a different point on the frontier. Finally, a post-hoc checkpoint audit shows that nearly identical 2K validation curves can conceal a 6.70-nat difference at 256K. Because those runs were neither seed-paired nor randomized across devices, we interpret this as checkpoint variability associated with an infrastructure transition, not a causal hardware effect. Code: https://github.com/kreasof-ai/atma
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。