Hydra通过模块化设计实现长文本推理的高效计算，显著提升速度与准确率。
Hydra: A Modular Architecture for Efficient Long-Context Reasoning
- 采用状态空间模型骨架，动态切换稀疏注意力、专家混合和双记忆机制
- 在8K上下文下速度提升3.01倍，多步逻辑推理准确率提高10倍
- 适合资源受限场景下的长文本理解任务，如复杂推理与知识检索
Transformer的二次复杂度从根本上限制了推理系统在资源受限和长上下文场景中的部署。我们提出Hydra，一种基于状态空间模型的模块化架构，可自适应地在互补的高效机制间路由：稀疏全局注意力、专家混合模型以及包含推理工作区与产品键记忆的双记忆结构。我们评估了一个2900万参数模型在合成序列上的逻辑链准确率与吞吐量，以及在WikiText上的吞吐量表现。消融实验使用特定合成数据集隔离各组件贡献。Hydra在8K token长度下，合成数据集和WikiText分别实现3.01倍和3.0倍的吞吐提升，并在多步逻辑组合任务中相较同规模Transformer实现10倍准确率提升。消融结果确认各组件作用：稀疏注意力捕捉长程依赖，专家模型针对输入领域专精，产品键记忆支持选择性检索。
原文摘要 · Abstract (English)
The quadratic complexity of transformers fundamentally limits reasoning system deployment in resource-constrained and long-context settings. We introduce Hydra, a modular architecture based upon a state-space backbone which adaptively routes between complementary efficiency mechanisms: sparse global attention, mixture-of-experts, and dual memories comprising a reasoning workspace and product key memory. We evaluate a 29M parameter model measuring logical chaining accuracy and throughput on synthetic sequences, plus throughput on WikiText. Ablation studies use component-specific synthetic datasets to isolate individual mechanisms. Hydra achieves $3.01\times$ and $3.0\times$ throughput gains at 8K tokens for synthetic and WikiText datasets, respectively, and $10\times$ accuracy improvements on multi-step logical composition compared to equal-sized transformers. Ablations confirm each component's contribution: sparse attention captures long-range dependencies, experts specialize to input domains, and product key memory enables selective retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。