用强化学习动态调度模型，省钱又提效。
Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning
- 训练小策略网络，按推理步骤自动选模型大小
- 在三个数学数据集上精度提升，成本比手工方法低30%以上
- 无需大奖励模型，适合资源有限的部署场景
推理时计算显著提升了大语言模型在复杂推理任务上的表现，但会带来高昂的推理成本。一种解决方案是将中间思维链（CoT）状态路由到不同规模的语言模型；然而，现有方法依赖人工设计的路由策略，限制性能，或需训练大型过程奖励模型，许多应用难以实现。本文将分步模型路由建模为约束决策问题，通过强化学习训练小型控制策略，并结合阈值校准调节性能与效率的权衡。我们在三个数学基准（GSM8K、MATH500、OmniMath）上对开源与闭源模型进行了验证。结果表明，该方法在准确率-成本权衡上持续优于手工策略，且达到与需训练大型过程奖励模型方法相当的水平。
原文摘要 · Abstract (English)
Inference-time computation has greatly enhanced the performance of large language models (LLMs) on challenging reasoning tasks, but this strategy can incur high inference costs. One solution is to route intermediate chain-of-thought (CoT) states to language models of different sizes; however, existing approaches rely on handcrafted routing strategies that limit performance, or on training large process reward models that may be infeasible in many applications. We formulate stepwise model routing as a constrained decision-making problem, which we solve by training a small control policy using reinforcement learning in conjunction with threshold calibration to tune the performance-efficiency tradeoff. We validate our method on three math benchmarks (GSM8K, MATH500, and OmniMath) on both open and closed models. Our method consistently improves the accuracy-cost tradeoff compared to handcrafted approaches, while achieving a comparable tradeoff to methods that require training large process reward models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。