用隐空间几何自动判断推理对错,训练速度提升2倍以上
Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clustering
- 从隐空间几何结构中挖掘内在奖励信号
- 正确推理路径形成密集聚类,错误路径为离散异常点
- 无需外部验证,适合高效训练大模型推理能力
组相对策略优化(GRPO)显著提升了大语言模型的推理能力,但其成功高度依赖昂贵的外部验证器或人工规则。这种依赖不仅带来高昂计算成本和训练延迟,还导致奖励稀疏,影响优化效率。为此,我们提出隐空间GRPO(Latent-GRPO),直接从隐空间几何结构中提取内在奖励。实证分析揭示:正确推理轨迹的终态标记表示形成高内聚性密集聚类,而错误轨迹则呈散乱分布。基于此发现,我们设计了迭代鲁棒中心估计(IRCE)算法,通过球面投影缓解幅度波动,并通过迭代聚合估计鲁棒的“真理中心”。在多个数据集上的实验表明,该方法在保持模型性能的同时,训练速度比基线提升2倍以上。大量结果还证明其具备强泛化能力和鲁棒性。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) significantly enhances the reasoning performance of Large Language Models (LLMs). However, this success heavily relies on expensive external verifiers or human rules. Such dependency not only leads to significant computational costs and training latency, but also yields sparse rewards that hinder optimization efficiency. To address these challenges, we propose Latent-GRPO, a framework that derives intrinsic rewards directly from latent space geometry. Crucially, our empirical analysis reveals a compelling geometric property: terminal token representations of correct reasoning trajectories form dense clusters with high intra-class similarity, whereas incorrect trajectories remain scattered as outliers. In light of this discovery, we introduce the Iterative Robust Centroid Estimation (IRCE) algorithm, which generates dense, continuous rewards by mitigating magnitude fluctuations via spherical projection and estimating a robust ``truth centroid'' through iterative aggregation. Experimental results on multiple datasets show that our method maintains model performance while achieving a training speedup of over 2x compared to baselines. Furthermore, extensive results demonstrate strong generalization ability and robustness. The code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。