模型测试时计算资源分配不智能，难应对多题统筹决策。
Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions
- 设计考试式评估框架，模拟共享计算预算下的多题求解
- 多数模型按题目顺序优先处理，忽视难度与分值差异
- 即使加提示也难实现价值与难度感知，适合研究推理效率者关注
推理语言模型越来越依赖测试时计算来提升表现，但现有评估多为单题独立进行。当多个问题共享总计算成本或延迟约束时，模型需决定如何在不同题目间分配有限的推理计算资源。本文提出一种类考试评估框架，要求模型在统一的令牌预算下，根据题目难度和分值分布计算资源以最大化总得分。实验覆盖多个开放与前沿推理模型，发现它们无法在不同难度与分值题目间进行策略性分配：行为近乎贪心顺序求解，优先处理靠前题目，且对题目价值不敏感，该倾向随题目数量增加而加剧。尽管显式规划提示可使计算分布更均匀，但仍未实现基于价值或难度的智能排序。该现象在数学与代码推理任务中均存在。结果表明，全局预算分配是一项尚未被传统评估覆盖的独立能力，仍是当前推理模型的挑战。
原文摘要 · Abstract (English)
Reasoning language models increasingly use test-time compute to improve performance, but existing evaluations typically study this compute one question at a time. Yet when multiple problems share an end-to-end cost or latency constraint, models must decide how to divide limited inference compute among them. We introduce an exam-style evaluation framework for studying this setting, in which a model must distribute one shared token budget across questions with different difficulty and point values to maximize its total score. Across several open and frontier reasoning models, we find that models fail to allocate a shared budget strategically across questions of varying difficulties and values. Models behave largely as greedy sequential solvers: they prioritize questions by presentation order, front-load effort on early questions, and remain insensitive to value, with these tendencies becoming more pronounced as the number of questions grows. Explicit planning prompts spread compute more evenly but do not produce value- or difficulty-aware prioritization. The same behavioral pattern extends from mathematical to code reasoning. These findings establish global budget allocation as a distinct capability that is not captured by conventional per-question evaluation and remains a challenge for current reasoning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。