Harmonic通过分层状态空间模型实现长文本高效建模，突破注意力机制的长度限制。
Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling
- 采用三级递进时序的分层状态空间结构，用误差反馈替代原始隐藏状态传递。
- 在32K上下文长度上比Transformer低11.4%的困惑度，64K时仍可训练成功。
- 适合需要超长序列建模且资源受限的场景，尤其适用于替换现有大模型的位置编码模块。
我们提出Harmonic，一种用于语言建模的分层状态空间模型（SSM）。该架构在逐步减慢的时间尺度上堆叠三个循环层级；每一层级接收下一层的预测误差作为输入，而非原始隐藏状态。在enwiki8数据集上，同等参数预算下，Harmonic在1K tokens时优于对比Transformer（28M参数）1.4%，在8K时提升6.7%，在32K时提升11.4%（bpt越低越好）。其性能也全面超越Mamba，在各测试长度上均领先0.7–1.8%。在64K tokens时，Mamba与Transformer因内存不足在80GB H100上无法运行，而Harmonic成功训练，达到6.169 bpt。结果在WikiText-103上复现，H-TF差距在1K–32K范围内为+1.7%至+7.2%。在10亿参数规模下，将TinyLlama 1.1B中的所有注意力层替换为HarmonicBlock后，消除了RoPE位置编码的长度限制：Hallamonic模型在两个独立干净基准（Lambada和fineweb-edu held-out）上保持稳定损失，序列长度1K–8K内表现良好，而TinyLlama在超过2K后崩溃（在Lambada上8K时相差+9.4 bpt）。计算复杂度为每前向传播O(L)，相比注意力的O(L²)显著降低。
原文摘要 · Abstract (English)
We present Harmonic, a hierarchical state space model (SSM) for language modeling. The architecture stacks three recurrent levels at progressively slower timescales; each level receives the prediction error of the level below as input, rather than its raw hidden state. On enwiki8 with equal token budgets, Harmonic outperforms a comparable Transformer (28M params) by +1.4% at 1K tokens, +6.7% at 8K tokens, and +11.4% at 32K tokens (bpt, lower is better). It also outperforms Mamba at every tested length by 0.7--1.8%. At 64K tokens, both Mamba and Transformer run out of memory on an 80GB H100; Harmonic trains successfully, reaching 6.169 bpt. Results replicate on WikiText-103 (H-TF gap +1.7% to +7.2% across 1K--32K). At 1B parameter scale, replacing all attention layers in TinyLlama 1.1B with HarmonicBlock eliminates the RoPE positional encoding limit: the resulting Hallamonic model maintains stable loss across sequence lengths 1K--8K on two independent clean benchmarks (Lambada and fineweb-edu held-out), while TinyLlama degrades catastrophically past its 2K-token RoPE limit (gap: +9.4 bpt at seq=8K on Lambada). Compute is O(L) per forward pass vs. O(L^2) for attention. Logs: https://github.com/Omibranch/harmonic-logs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。