用梯度方向选题，让大模型强化学习更稳定高效
GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning
- 根据验证集梯度方向筛选训练问题，动态构建优化课程
- 在奖励信号不可靠等场景下，性能超越现有方法
- 适合追求稳定训练和高精度的LLM强化学习研究者
强化学习（RL）已成为大语言模型（LLMs）后训练的核心范式，但其性能对训练问题质量高度敏感。这种敏感性源于RL的非平稳性：轨迹由不断演化的策略生成，学习受探索与奖励反馈影响，不同于具有固定轨迹的监督微调（SFT）。因此，以往工作常依赖人工标注或简单启发式过滤（如准确率），易引入错误或低效问题。我们提出GradAlign，一种基于梯度对齐的数据选择方法，利用少量可信验证集，优先选择其策略梯度与验证梯度方向一致的训练问题，实现自适应课程学习。我们在三种挑战性数据设置下评估：不可靠的奖励信号、分布偏移和低效训练语料，结果表明GradAlign始终优于现有基线，凸显了方向性梯度信号在非平稳策略优化中的关键作用，带来更稳定的训练过程和更高的最终性能。代码已开源：https://github.com/StigLidu/GradAlign
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a central post-training paradigm for large language models (LLMs), but its performance is highly sensitive to the quality of training problems. This sensitivity stems from the non-stationarity of RL: rollouts are generated by an evolving policy, and learning is shaped by exploration and reward feedback, unlike supervised fine-tuning (SFT) with fixed trajectories. As a result, prior work often relies on manual curation or simple heuristic filters (e.g., accuracy), which can admit incorrect or low-utility problems. We propose GradAlign, a gradient-aligned data selection method for LLM reinforcement learning that uses a small, trusted validation set to prioritize training problems whose policy gradients align with validation gradients, yielding an adaptive curriculum. We evaluate GradAlign across three challenging data regimes: unreliable reward signals, distribution imbalance, and low-utility training corpus, showing that GradAlign consistently outperforms existing baselines, underscoring the importance of directional gradient signals in navigating non-stationary policy optimization and yielding more stable training and improved final performance. We release our implementation at https://github.com/StigLidu/GradAlign
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。