通过几何一致性检测修复噪声奖励，提升大模型强化学习稳定性。
GEOALIGN: Geometric Rollout Curation for Robust LLM Reinforcement Learning
- 构建提示内偏好对，用在线投影器聚焦奖励方向
- 识别并修正与多数方向冲突的异常生成路径，降低训练波动
- 轻量插件式设计，适合对话对齐与数学推理任务
在线强化学习广泛用于对齐大语言模型与奖励信号，但在噪声或错误奖励下训练易不稳定。我们发现一种称为方向不一致的失败模式：一批生成结果中，少数高奖励样本会诱导出与整体多数严重偏离的表示空间偏好方向，导致高方差和训练崩溃。本文提出geoalign，一种轻量级迭代策略优化中的生成路径筛选方法。geoalign（i）在单个提示内构建偏好对，（ii）对每条生成路径的隐藏状态学习一个在线投影器，集中奖励有序的位移方向，（iii）通过计算路径方向与批次共识原型的夹角偏差，检测方向不一致的路径，并用提示内的稳定替代路径进行修正。该方法仅需前向传播，开销极小。在基于学习奖励模型的对话对齐与基于二值验证奖励的数学推理任务中，geoalign均提升最终性能并显著减少训练震荡，优于PF-PPO、PAR、PODS和Seed-GRPO。结果表明，潜在的方向一致性可作为在线大模型强化学习的有效可靠性信号。
原文摘要 · Abstract (English)
Online reinforcement learning is widely used to align large language models (LLMs) with reward signals, yet training can be unstable under noisy or misspecified rewards. We identify a failure mode we call directional inconsistency: within a batch, a small set of high-reward rollouts induces representation-space preference directions that sharply disagree with the batch majority, resulting in high-variance and destabilizing updates. We propose geoalign, a lightweight plug-in for rollout curation in iterative policy optimization. Geoalign (i) forms within-prompt preference pairs, (ii) learns an online projector on per-rollout hidden states to concentrate reward-ordered displacement directions, and (iii) detects directionally inconsistent rollouts via their angular deviation from a batch consensus prototype and rectifies them with within-prompt stable alternatives. Geoalign is forward-pass only and adds negligible overhead. Across dialogue alignment with a learned reward model and mathematical reasoning with binary verified rewards, Geoalign improves final performance and reduces training oscillation, outperforming PF-PPO, PAR, PODS, and Seed-GRPO. These results suggest latent directional consensus as an effective reliability signal for online LLM RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。