arXiv:2602.10819cs.LG2026-02

让大模型在不破坏稳定性前提下,学会处理难样本。

RePO: Bridging On-Policy Learning and Off-Policy Knowledge through Rephrasing Policy Optimization

  • 用重述策略把外部知识转成模型能吸收的推理路径
  • 在多个基准上提升难样本利用率,达到当前最佳效果
  • 适合需要强化学习但又怕训练不稳的研究者

将大语言模型(LLMs)对领域数据的对齐仍是一个基本挑战。监督微调(SFT)虽可注入领域知识,但常损害模型通用性;而基于策略的强化学习(RL)虽保持通用性,却难以有效吸收超出当前推理能力的难样本。近期的非策略强化学习尝试提升了难样本利用效率,但因强制分布偏移导致严重训练不稳定。为兼顾非策略知识吸收与策略学习的稳定性,本文提出重述策略优化(RePO)。在 RePO 中,模型被提示先理解非策略知识,再将其重述为符合自身风格与参数分布的轨迹。系统动态以高质量重述轨迹替换低奖励采样路径,引导模型走向正确推理路径,同时严格维持策略学习的训练动态。在多个基准测试中,RePO 显著提升难样本利用率,性能超越现有基线,达到当前最优水平。

原文摘要 · Abstract (English)

Aligning large language models (LLMs) on domain-specific data remains a fundamental challenge. Supervised fine-tuning (SFT) offers a straightforward way to inject domain knowledge but often degrades the model's generality. In contrast, on-policy reinforcement learning (RL) preserves generality but fails to effectively assimilate hard samples that exceed the model's current reasoning level. Recent off-policy RL attempts improve hard sample utilization, yet they suffer from severe training instability due to the forced distribution shift toward off-policy knowledge. To reconcile effective off-policy knowledge absorption with the stability of on-policy RL, we propose Rephrasing Policy Optimization (RePO). In RePO, the policy model is prompted to first comprehend off-policy knowledge and then rephrase it into trajectories that conform to its own stylistic and parametric distribution. RePO dynamically replaces low-reward rollouts with these rephrased, high-quality trajectories. This strategy guides the model toward correct reasoning paths while strictly preserving on-policy training dynamics. Experiments on several benchmarks demonstrate that RePO improves hard-sample utilization and outperforms existing baselines, achieving state-of-the-art performance.

强化学习大模型对齐知识吸收

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。