根据错误后果分配推理计算,让关键任务获得更高算力。
Not All Errors Are Equal: Consequence-Aware Reasoning Compute Allocation

- 用轻量级预测器评估任务出错的代价,指导算力分配。
- 在相同预算下,代价加权损失降低22%至33%。
- 适合对故障成本敏感的生产级软件部署场景。
现代推理模型可在测试时动态分配计算资源(如思考令牌、模型调用或算力预算)以应对不同任务。现有方法通常依据任务预估难度分配算力,认为更难的任务需更多计算以提升准确率。但这隐含了所有失败代价相同的假设,而实际部署中,日志拼写错误与生产数据库迁移导致的数据损坏虽在基准中均计为一次失败,但真实影响截然不同。为此,我们提出后果感知的测试时算力分配机制:不只根据难度,还通过轻量级预测器从问题文本估计若任务失败将带来的代价,调度系统据此将高后果任务分配至更高算力层级或更大思考预算。我们在SWE-bench Lite上进行主实验,并在Multi-SWE-bench mini上评估跨数据集表现,共覆盖700个软件工程任务。结果表明,后果与难度在多种标注下近似正交,当前模型未充分按后果分配算力。此外,我们的仅基于问题文本的预测器在300个SWE-bench任务中从未将高后果任务误判为低后果。在匹配算力预算下,我们的后果感知调度器相较难度感知路由,代价加权损失减少22%至33%;其中优先级感知变体(按每任务代价乘以边际效用信号调度)超过30%,其可部署的预测器驱动版本仍保留超90%的最优收益。
原文摘要 · Abstract (English)
Modern reasoning models can allocate different amounts of test-time computation, such as thinking tokens, model calls, or compute budget, to different tasks. Existing methods generally drive this allocation by predicted difficulty and spend more compute where it is expected to raise accuracy. This implicitly assumes that all failures cost the same, since an accuracy objective weights every task equally. However, such an assumption does not hold in deployment: A typo in a log message and a migration that corrupts a production database both count as one benchmark failure, but their real-world costs are fundamentally different. To fill this gap, we propose consequence-aware test-time compute allocation. Instead of routing compute only by predicted difficulty, we use a lightweight predictor to estimate from the issue text how costly a task would be if solved incorrectly. The scheduler then routes higher-consequence tasks to larger compute tiers or higher thinking budgets under the same total budget. We conduct main experiments on SWE-bench Lite and evaluate cross-dataset behavior on Multi-SWE-bench mini, covering 700 software-engineering tasks in total. Our results reveal that consequence and difficulty are approximately orthogonal under various annotations, and that current thinking models do not allocate compute sufficiently according to consequence. Moreover, our issue-only predictor never misclassifies a high-consequence task as low-consequence across the 300 SWE-bench tasks. Under matched compute budgets, our consequence-aware scheduler reduces cost-weighted loss by 22% to 33% relative to difficulty-aware routing; in particular, the priority-aware variant, which routes by per-task cost scaled by the marginal-utility signal, crosses 30%, and its deployable predictor-driven version retains over 90% of the oracle gain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。