提出并行混合架构，用分治方式提升长文本建模效率与精度。
Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing
- GSS、GQA、FFN三支路并行处理，通过可学习混合机制融合
- 125M参数下在WikiText-103上达16.51困惑度，优于多个基线
- 相比纯注意力模型，吞吐量高24%，内存占用低40%
长序列建模仍是自然语言处理的核心挑战。Transformer依赖自注意力，但复杂度随序列长度呈二次增长（$O(N^2)$），而状态空间模型（SSMs）虽线性扩展（$O(N)$），却存在选择性记忆瓶颈，难以从压缩状态中精准召回信息。为此，我们提出并行混合架构（PHA），将门控状态空间（GSS）、分组查询注意力（GQA）和前馈网络（FFN）作为独立并行分支，通过可学习混合机制融合。各分支各司其职：GSS捕获全局上下文，注意力执行选择性检索，FFN提供补充处理。在WikiText-103上，PHA以125M参数实现16.51困惑度，优于Hedgehog（16.70）和H3-125M（23.70）。扩展至180M参数时困惑度降至16.42，性能媲美纯注意力模型，同时在长序列下实现24%更高吞吐量和最高40%更低内存消耗。在OpenWebText上，125M模型达19.72困惑度，超越标准Transformer（20.60）及GSS基线（19.80）。结果表明，将不同建模范式拆分为并行专家，可在保持Transformer级困惑度的同时显著提升长序列建模效率。
原文摘要 · Abstract (English)
Modeling long-range dependencies remains a central challenge in natural language processing. Transformer architectures achieve strong performance via self-attention but scale quadratically ($O(N^2)$) with sequence length, while State Space Models (SSMs) scale linearly ($O(N)$) but suffer from a selective recall bottleneck, struggling to retrieve precise information from compressed states. This creates a fundamental tradeoff between efficiency and perplexity. To tackle these challenges, we propose the \textit{Parallel Hybrid Architecture (PHA)}, which runs Gated State Spaces (GSS), Grouped Query Attention (GQA), and Feed-Forward Networks (FFNs) as independent parallel branches fused by a learnable mixing mechanism. Instead of forcing SSMs to approximate attention or serializing the two paradigms, PHA allows each branch to specialize: GSS captures global context, while attention performs selective retrieval, with FFN providing complementary processing. On WikiText-103, PHA achieves 16.51 PPL at 125M parameters, outperforming Hedgehog (16.70) and H3-125M (23.70). Scaling to 180M parameters yields 16.42 PPL, which gives comparable results with the pure attention baseline while delivering 24\% higher throughput and up to 40\% lower memory usage at long contexts. On OpenWebText, our 125M model achieves 19.72 PPL, outperforming standard Transformers (20.60) and GSS hybrid baselines (19.80). These results demonstrate that separating sequence modeling paradigms into parallel specialists enables Transformer-level perplexity with substantially improved efficiency for long-context language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。