动态分配推理预算，让大模型自适应调整更省计算且准确
OptPO: Optimal Rollout Allocation for Test-time Policy Optimization
- 用贝叶斯序贯检验决定何时停止生成推理路径
- 相比固定采样减少近半推理开销，准确率不降反升
- 适合需要高效测试时优化的推理系统开发者
测试时策略优化使大语言模型能通过自生成推理路径反馈来适应分布偏移。但现有方法依赖固定预算的多数投票估计奖励，造成大量计算冗余。本文提出最优推理路径分配框架 OptPO，将投票过程建模为贝叶斯序贯概率比检验，动态停止采样直至后验置信度达到阈值。关键优势在于：保留的推理路径可直接用于策略更新，无缝集成 PPO 或 GRPO 等算法，无需真实标签。在多个推理基准上，OptPO 显著降低推理开销，同时保持或提升准确率。通过统一统计最优停止与测试时学习，OptPO 提供了一种计算高效的测试时适应新范式。代码将在录用后开源。
原文摘要 · Abstract (English)
Test-time policy optimization enables large language models (LLMs) to adapt to distribution shifts by leveraging feedback from self-generated rollouts. However, existing methods rely on fixed-budget majority voting to estimate rewards, incurring substantial computational redundancy. We propose Optimal Rollout Allocation for Test-time Policy Optimization (OptPO), a principled framework that adaptively allocates inference budgets. By formulating the voting process as a Bayesian sequential probability ratio test, OptPO dynamically halts sampling once the posterior confidence in a consensus answer exceeds a specified threshold. Crucially, it utilizes the retained rollouts for on-policy updates, seamlessly integrating with algorithms like PPO or GRPO without requiring ground-truth labels. Across diverse reasoning benchmarks, OptPO significantly reduces rollout overhead compared to fixed-sample baselines while preserving or improving accuracy. By unifying statistically optimal stopping with test-time learning, OptPO offers a computationally efficient paradigm for test-time adaptation. The source code will be open upon acceptance at https://open-upon-acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。