对比大模型架构,发现推理增强远比混合结构更重要。
Reasoning Primitives in Hybrid and Non-Hybrid LLMs: Do Architectural Differences Yield Advantages in State-Tracking and Recall?
- 用五类状态追踪任务测试模型,对比纯变换器与混合结构
- 加入推理标记后,性能显著提升,最高差距达数倍
- 混合结构优势仅在特定任务中出现,更多影响推理效率
大型语言模型的推理常被视为单一能力,但其优势可能源于更基础的操作。我们通过五组受控任务,聚焦状态追踪与回忆能力,比较匹配的Transformer与混合架构在有无推理增强下的表现。结果显示,加入推理增强的变体显著优于仅依赖指令的版本,差距常达数倍。该结果支持“状态优于标记”观点:外部化推理痕迹能将中间状态有效传递至标记空间。相比之下,混合归纳偏置在推理标记可用时并未带来一致性能优势。当架构差异显现时,其表现取决于任务结构:混合思维模型在严格顺序链式更新任务中更稳健,而变换器思维模型在扁平多跳检索任务中表现更优。因此,本研究的核心贡献是揭示状态记忆任务中的性能驱动因素:推理标记增强是主导因素,而混合结构的优势更窄、依赖任务且可能更多体现为推理效率而非整体能力。我们还公开了代码与数据以供复现。
原文摘要 · Abstract (English)
Reasoning in large language models is often discussed as a single capability, but some of its gains may stem from simpler underlying operations. We examine two such primitives, recall and state-tracking, through five controlled task families centered on state-based recall, and compare matched transformer and hybrid architectures with and without reasoning augmentation. Across the suite, reasoning-augmented variants substantially outperform instruction-only variants, often by large margins. This pattern is consistent with the State over Tokens view: externalized reasoning traces help because they carry the intermediate state forward in token space. By contrast, hybrid inductive bias does not yield a uniform advantage in accuracy once reasoning tokens are available. When architectural differences do appear, they follow task structure: the hybrid Think model is more robust on strictly sequential chained updates, whereas the transformer Think model is more robust on flat multi-hop retrieval. We therefore cast the main contribution of this study as a descriptive account of what drives performance on state-based recall tasks: reasoning-token augmentation appears to be the dominant factor, while hybrid advantages are narrower, task-dependent, and potentially more about inference efficiency than overall capability. We also release the codebase and data required to reproduce these results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。