提出统一框架评测大模型思维模式切换策略,助力高效推理研究。
HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs

- 构建12种可控实验设置,覆盖三类切换策略与四类训练方式。
- 实测6个模型在5个基准上表现,发现策略效果随模型规模和任务变化。
- 提供开源代码与数据,支持可复现的推理效率研究。
混合推理大语言模型允许显式控制推理开销,实现答案质量与计算成本的权衡。然而,现有自适应思维模式选择方法常在不同模型、数据集和实现假设下评估,难以横向比较。本文提出HRBench,一个统一评估框架,用于研究混合推理模型中的思维模式切换。该框架沿两个维度组织:三种切换策略家族——基于提示的选择、外部路由、推测执行;四种训练范式——无训练、SFT、离线RL、在线RL,共形成12种受控评估场景。我们在6个模型(从Qwen3.5-2B到Kimi-K2.5-1.1T)和5个涵盖数学、科学、代码的推理基准上评估这些设置,并在同一流程中重实现12+种代表性前期方法。分析表明,不同策略占据不同的有效性-效率权衡区域:基于提示的方法通常具有更优的令牌-准确率权衡,路由方法带来更稳定的成本降低,推测方法则在更高令牌开销下提升准确率。进一步发现,训练方式对策略影响各异,最优策略随模型规模和任务领域而变。HRBench提供参考实现与统一评估平台,推动高效推理研究的可控性发展。数据、代码与仓库已公开于https://github.com/usail-hkust/HRBench。
原文摘要 · Abstract (English)
Hybrid-reasoning large language models (LLMs) expose explicit controls over reasoning effort, allowing users or systems to trade off answer quality against inference cost. However, existing methods for adaptive thinking-mode selection are typically evaluated under different models, datasets, and implementation assumptions, making it difficult to compare their practical behavior. We introduce HRBench, a unified evaluation framework for studying thinking-mode switching in hybrid-reasoning LLMs. HRBench organizes the design space along two axes: three switching strategy families, prompt-based selection, external routing, and speculative execution, and four training regimes, training-free, SFT, offline and online RL, yielding 12 controlled evaluation settings. We evaluate these settings across 6 LLMs, from Qwen3.5-2B to Kimi-K2.5-1.1T, and 5 reasoning benchmarks covering mathematics, science, and code, while reimplementing 12+ representative prior methods within the same pipeline. Our analysis characterizes how different switching strategies occupy distinct effectiveness-efficiency trade-off regions: prompt-based methods often provide favorable token-accuracy trade-offs, routing methods offer more stable cost reduction, and speculative methods tend to improve accuracy at higher token cost. We further find that training affects strategies differently, and that the preferred strategy varies with model scale and task domain. HRBench provides reference implementations and a unified evaluation platform to support more controlled research on efficient reasoning in hybrid-reasoning LLMs. Our data, code and repository are available at https://github.com/usail-hkust/HRBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。