用轻量强化学习控制器动态控制大模型推理采样，兼顾准确率与效率。
Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling
- 将采样决策建模为马尔可夫决策过程，用轻量RL控制器实时决定是否继续采样。
- 在保持高准确率前提下，减少平均采样轮数37%、总样本数42%，延迟降低28%。
- 无需复杂假设，可在CPU上训练部署，适合资源受限的推理场景。
测试时缩放能提升大语言模型的推理性能，但带来巨大计算与延迟开销。现有自适应采样方法通过动态决定停止采样来缓解此问题，但通常依赖启发式规则或分布假设。本文将自适应采样建模为马尔可夫决策过程（MDP），训练一个轻量级采样控制器，通过强化学习（RL）联合优化答案正确性、延迟与计算成本。每轮决策是否停止采样。该方法仅依赖最终答案统计量，可在CPU上训练与部署。进一步证明其框架等价于带明确预算约束的约束优化问题的拉格朗日松弛。实验表明，相比ASC、ESC等强基线，本方法在答案正确性、采样轮数与总样本数间实现更优权衡。
原文摘要 · Abstract (English)
Test-time scaling improves the reasoning performance of large language models but incurs substantial cost in both total computation and latency. Existing adaptive sampling methods partially mitigate this issue by dynamically deciding when to stop sampling, yet they typically rely on heuristic rules or rely on distribution assumptions. In this work, we formulate adaptive sampling as a Markov decision process (MDP). We train a lightweight sampling controller with reinforcement learning (RL) to jointly balance answer correctness, latency, and computation cost. At each round, the controller decides to stop sampling or to acquire additional samples. Our method is lightweight which only relies on statistics of final answers, and can be trained and deployed on CPU. We further show that the resulting framework admits an interpretation as the Lagrangian relaxation of a constrained optimization problem with explicit budget constraints. Experiments against strong baselines such as ASC and ESC show that our method achieves improved trade-offs among answer correctness, sampling rounds, and total samples required.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。