通过深度注意力混合提升模型推理效率，显著降低训练成本。
Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves
- 引入深度循环注意力混合框架，跨层共享参数并解耦缩放维度。
- 相同精度下训练量减少2至8倍，性能优于更大规模的现有模型。
- 揭示深度间知识使用差异，专家选择多样性提升2至11倍。
深度循环通过跨深度共享参数促进潜在推理，但以往工作缺乏在计算量、参数量和内存上匹配的基线，部分固定层数限制了深度循环的利用，并忽视了恒定隐状态尺寸对多步推理的瓶颈。为此，我们提出深度循环注意力混合（Dreamer）模块化框架，融合序列注意力、深度注意力与稀疏专家注意力。该方法通过深度方向的注意力缓解隐状态尺寸瓶颈，解耦缩放维度，使深度循环模型可高效且有效扩展。在语言推理基准测试中，我们的模型在与最优模型同等计算量、参数量和内存条件下，所需训练标记数仅为后者的2至8倍，且在相同训练标记数下性能超越约2倍大的现有最优模型。此外，我们揭示了深度间的知识使用模式，例如发现专家选择多样性较现有MoE高出2至11倍。
原文摘要 · Abstract (English)
Depth-recurrence facilitates latent reasoning by sharing parameters across depths. However, prior work lacks combined FLOP-, parameter-, and memory-matched baselines, underutilizes depth-recurrence due to partially fixed layer stacks, and ignores the bottleneck of constant hidden-sizes that restricts many-step latent reasoning. To address this, we introduce a modular framework of depth-recurrent attention mixtures (Dreamer), combining sequence attention, depth attention, and sparse expert attention. It alleviates the hidden-size bottleneck through attention along depth, decouples scaling dimensions, and allows depth-recurrent models to scale efficiently and effectively. Across language reasoning benchmarks, our models require 2 to 8x fewer training tokens for the same accuracy as FLOP-, parameter-, and memory-matched SOTA, and outperform ca. 2x larger SOTA models with the same training tokens. We further present insights into knowledge usage across depths, e.g., showing 2 to 11x larger expert selection diversity than SOTA MoEs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。