让大模型推理更高效，通过动态重用经验提升强化学习效果
Experience Augmented Policy Optimization for LLM Reasoning
- 用已有策略作为经验先验，在关键节点选择性注入
- 在5个基准上优于现有最强RLVR方法，性能持续提升
- 适合研究大模型推理优化与强化学习的开发者
基于可验证奖励的强化学习（RLVR）是提升大语言模型（LLM）推理能力的有效方法。然而，现有RLVR方法通常从零开始进行在线策略优化，导致采样成本高，且难以有效利用累积经验。随着模型能力与策略行为的变化，现有方法通过固定推理轨迹重用经验，进一步引发策略不匹配问题。本文提出经验增强型策略优化（EAPO），不再将经验视为固定推理路径，而是以预训练的强化学习策略作为动作级经验先验，并在推理过程中选择性地在关键决策点注入经验。为确保从经验增强的推演中稳定、无偏地学习，EAPO引入改进的重要性采样机制。在Qwen-2.5-math-7b和Qwen-3-8B两个模型上，于五个不同基准上的实验表明，EAPO持续优于当前最先进的RLVR方法。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing RLVR methods typically rely on on-policy optimization from scratch, resulting in high sampling costs and inefficient utilization of accumulated experience. As model capabilities and policy behaviors evolve during training, recent attempts to reuse experience via fixed reasoning trajectories further suffer from policy mismatch. Motivated by these limitations, we argue that experience in RLVR should not be reused as fixed reasoning trajectories, but instead expressed in a policy-adaptive manner. In this work, we propose Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout. To ensure stable and unbiased learning from experience-augmented rollouts, EAPO further incorporates an adapted importance sampling scheme. Experiments on using Qwen-2.5-math 7b and Qwen-3-8B on five different benchmarks demonstrate that EAPO consistently improves reasoning performance over state-of-the-art RLVR methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。