通过识别无效推理轨迹，提升PPO微调效率与模型可靠性
Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training
- 基于梯度近似计算每条轨迹的影响力得分，筛选有效训练数据
- 在多个任务上超越SFT和传统PPO，减少错误思维链达37%
- 适合需要高质量推理的LLM后训练场景，如复杂问答系统
传统强化学习算法（如近端策略优化，PPO）通常使用整个回放缓冲区进行训练，假设所有生成的轨迹均提供有益的优化信号。然而这些轨迹常包含噪声或不忠实的推理过程，导致模型性能下降并减缓训练速度。本文提出一种新框架——影响引导的PPO（I-PPO），将数据归因机制融入强化学习后训练流程。通过梯度基近似计算每条轨迹的影响得分，I-PPO能够识别并剔除与验证梯度反向对齐的轨迹。实验表明，I-PPO在多个基准测试中持续优于监督微调（SFT）和标准PPO基线。我们的过滤机制展现出内在的早停效应，在加速训练效率的同时，有效降低不忠实思维链（CoT）比例达37%。
原文摘要 · Abstract (English)
Traditional RL algorithms like Proximal Policy Optimization (PPO) typically train on the entire rollout buffer, operating under the assumption that all generated episodes provide a beneficial optimization signal. However, these episodes frequently contain noisy or unfaithful reasoning, which can degrade model performance and slow down training. In this paper, we propose \textbf{Influence-Guided PPO (I-PPO)}, a novel framework that integrates data attribution into the RL post-training loop. By calculating an influence score for each episode using a gradient-based approximation, I-PPO identifies and eliminates episodes that are anti-aligned with a validation gradient. Our experiments demonstrate that I-PPO consistently outperforms SFT and PPO baselines. We show that our filtering process acts as an intrinsic early stopping mechanism, accelerating training efficiency while effectively reducing unfaithful CoT reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。