提出测试时扩展的三类推理范式,统一评估标准以提升可复现性。
Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility

- 区分单路径、叶级和前缀级三种测试时扩展方法
- 设计包含计算开销与不确定性估计的评估指标体系
- 适合关注大模型推理可靠性与实验可复现性的研究者
大语言模型可通过增加推理时计算资源解决更复杂的推理问题。然而,“测试时扩展”涵盖多种不同算法:沿单一轨迹延长思考、采样多个候选答案并投票或验证、在部分状态空间中搜索。这些方法在统计结构、计算需求和失败模式上存在差异。将它们视为单一标量“预算”下的可互换方案,或仅报告准确率而不说明推理协议,会使得跨研究结果难以比较。本文从三个维度研究测试时扩展:首先,将推理形式化为对自回归模型隐式前缀树的带预算推断,区分单路径顺序扩展、叶级终端缩减和前缀级扩展;其次,将完整推理系统作为评估对象,分离端到端性能与候选库诊断,提出评估轮廓,其坐标与简单函数可恢复或界定常见重复采样指标,并要求计算开销与不确定性估计与协议一致;第三,区分精确重放与分布可复现性并明确各自要求。同时,我们按模型侧与接口机制组织开源推理模型。实证研究覆盖广泛知识、符号推理与竞赛数学,共公开释放1,403,520次模型尝试数据。项目网站为 https://mohsenhariri.github.io/scorio/tts,数据集包括 Trace(https://huggingface.co/datasets/harimo/scorio-trace)、Lite(https://huggingface.co/datasets/harimo/scorio-lite)、Math(https://huggingface.co/buckets/harimo/scorio-math)和 SuperGPQA(https://huggingface.co/buckets/harimo/scorio-gpqa)。
原文摘要 · Abstract (English)
Large language models can solve harder reasoning problems with more inference-time compute. The term "test-time scaling," however, covers several inference algorithms: extending deliberation along one trajectory, sampling completed candidates and aggregating them by voting or verification, and searching over partial states. These algorithms differ in statistical structure, compute requirements, and failure modes. Treating them as interchangeable under a scalar "budget," or reporting accuracy without specifying the inference protocol, makes results difficult to compare across studies. We study test-time scaling along three axes. First, we formalize it as budgeted inference over the implicit prefix tree of an autoregressive model and distinguish single-trajectory sequential scaling, leaf-level scaling with terminal reduction, and prefix-level scaling. Second, we treat the full inference system as the evaluated object and separate end-to-end performance from candidate-bank diagnostics. We introduce an evaluation profile whose coordinates and simple functionals recover or bound common repeated-sampling metrics, and require compute accounting and uncertainty estimates that match the protocol. Third, we distinguish exact replay from distributional reproducibility and state the requirements for each. We also organize open-weight reasoning models by model-side and interface mechanisms. Our empirical study covers broad knowledge, symbolic reasoning, and competition mathematics, and we publicly release 1,403,520 sampled model attempts. The project website is available at https://mohsenhariri.github.io/scorio/tts. The released datasets are Trace (https://huggingface.co/datasets/harimo/scorio-trace), Lite (https://huggingface.co/datasets/harimo/scorio-lite), Math (https://huggingface.co/buckets/harimo/scorio-math), and SuperGPQA (https://huggingface.co/buckets/harimo/scorio-gpqa).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。