arXiv:2604.14853cs.LG2026-04被引 8

根据输入难易自动分配推理计算量,提升大模型准确率

Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization

论文配图:Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization
图 1 · 摘自论文原文
  • 用约束优化方法决定每题该花多少算力
  • 在数学题数据集上最高提升12.8%准确率
  • 适合资源有限时想高效使用大模型的用户

测试时计算扩展通过重复采样、搜索或延长推理来提升大语言模型性能。但在有限推理预算下,当前系统忽视了一个关键决策:哪些输入应分配更多计算,哪些可低成本回答?本文将此问题形式化为带约束的优化问题(在平均计算预算限制下最大化预期准确率),并提出两阶段求解-学习框架。求解阶段利用拉格朗日松弛将全局约束分解为每个实例的子问题,得到闭式最优动作(权衡准确率与成本);证明其成本随对偶变量单调,可用二分查找精确控制预算。学习阶段训练轻量分类器,从输入特征预测最优动作,实现实时部署。理论证明学习策略的任务级遗憾受模仿误差与最坏情况下单实例差距乘积控制,实现从约束推理到监督分类的简洁还原。在MATH和GSM8K数据集上,使用DeepSeek-V3、GPT-4o-mini、Qwen2.5-7B三个模型的实验表明,本方法持续优于均匀及启发式分配基线,在匹配预算条件下,于MATH上达到最高12.8%相对准确率提升,且以超过91%的模仿准确率紧追拉格朗日预言机上界。

原文摘要 · Abstract (English)

Test-time compute scaling, the practice of spending extra computation during inference via repeated sampling, search, or extended reasoning, has become a powerful lever for improving large language model performance. Yet deploying these techniques under finite inference budgets requires a decision that current systems largely ignore: which inputs deserve more compute, and which can be answered cheaply? We formalize this as a constrained optimization problem (maximize expected accuracy subject to an average compute budget) and solve it with a two-stage Solve-then-Learn pipeline. In the solve stage, Lagrangian relaxation decomposes the global constraint into per-instance sub-problems, each admitting a closed-form oracle action that optimally prices accuracy against cost. We prove that the induced cost is monotone in the dual variable, enabling exact budget targeting via binary search. In the learn stage, a lightweight classifier is trained to predict oracle actions from cheap input features, amortizing the allocation rule for real-time deployment. We establish that the task-level regret of the learned policy is bounded by its imitation error times the worst-case per-instance gap, yielding a clean reduction from constrained inference to supervised classification. Experiments on MATH and GSM8K with three LLMs (DeepSeek-V3, GPT-4o-mini, Qwen2.5-7B) show that our method consistently outperforms uniform and heuristic allocation baselines, achieving up to 12.8% relative accuracy improvement on MATH under matched budget constraints, while closely tracking the Lagrangian oracle upper bound with over 91% imitation accuracy.

推理优化大模型预算控制强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。