用对抗逆强化学习统一了强化学习与搜索，提升大模型推理性能。
AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- 通过参考轨迹自动推导密集奖励模型，无需人工标注过程数据。
- 在8个基准上平均提升9%，达到GPT-4o水平。
- 适用于数学、科学、代码生成等复杂推理任务，适合追求高效测试时扩展的团队。
大型语言模型的测试时扩展策略主要依赖基于结果的强化学习或静态过程奖励模型的搜索方法。然而，基于结果的强化学习常因训练不稳定和样本效率低而受限，静态过程奖励模型则需昂贵的逐步监督，且易受分布偏移影响导致奖励欺骗。本文提出AIRL-S框架，将对抗逆强化学习与组相对策略优化相结合。通过从参考轨迹中直接推断出密集的逐步奖励模型，该方法摆脱了对标注过程数据的依赖，并将同一学习到的奖励模型同时用于训练信号和搜索验证。在数学、科学和代码生成共八个基准上的实验表明，该策略使平均性能相比基础模型提升9%，达到GPT-4o水平。进一步分析揭示了AIRL与GRPO目标间的互补性，以及所学奖励模型在不同生成器和搜索算法间的迁移能力，建立了一种鲁棒且低成本的复杂推理任务测试时计算扩展方法。
原文摘要 · Abstract (English)
Test-time scaling strategies for Large Language Models predominantly rely on either reinforcement learning with sparse outcome rewards or search-based methods guided by static Process Reward Models. However, outcome-based RL often suffers from training instability and sample inefficiency, while static PRMs require expensive step-wise supervision and are susceptible to reward hacking due to distributional shifts. In this paper, we introduce AIRL-S, a unified framework that integrates Adversarial Inverse Reinforcement Learning with Group Relative Policy Optimization. By inferring a dense, step-wise reward model directly from reference trajectories, AIRL-S eliminates the dependency on labeled process data and uses the same learned PRM as both a training signal and a verifier for search-based TTS. Extensive evaluations across eight benchmarks in mathematics, science, and code generation demonstrate that our policy model improves average performance by 9\% over the base model, matching GPT-4o. We further analyze how the AIRL and GRPO objectives complement each other and how the learned PRM transfers across generators and search algorithms, establishing a robust and cost-effective methodology for scaling test-time computation in complex reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。