发现大模型强化学习效果差异源于概率分布的清晰度。
Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Models
- 用轮廓系数量化概率分布清晰度，揭示模型内部聚类与类别分离特性。
- 清晰度高的模型在数学推理任务上强化学习提升达5.9分，逻辑错误更少。
- 适合研究模型训练机制、强化学习优化或可解释性方向的读者。
语言模型家族在强化学习收益上存在显著差异：相同训练条件下，Qwen模型表现大幅提升，而Llama模型改进有限。我们发现这一差异源于隐藏的结构属性——概率空间中的分布清晰度。通过三阶段分析（现象到机制再到解释），揭示出具备强化学习友好性的模型，在正确与错误回答的概率分配上表现出类内紧凑性和类间分离性。我们使用轮廓系数（Silhouette Coefficient, $S$）进行量化，发现高$S$值与强化学习性能强相关，低$S$值则伴随严重逻辑错误和推理不稳定性。为验证该性质，提出一种轮廓感知重加权策略，优先处理低$S$样本。在六个数学基准测试中，所有模型家族均实现一致提升，最高达AIME24上的5.9分增益。本工作确立分布清晰度是强化学习友好性的一个基础且可训练的属性。
原文摘要 · Abstract (English)
Language model families exhibit striking disparity in their capacity to benefit from reinforcement learning: under identical training, models like Qwen achieve substantial gains, while others like Llama yield limited improvements. Complementing data-centric approaches, we reveal that this disparity reflects a hidden structural property: \textbf{distributional clarity} in probability space. Through a three-stage analysis-from phenomenon to mechanism to interpretation-we uncover that RL-friendly models exhibit intra-class compactness and inter-class separation in their probability assignments to correct vs. incorrect responses. We quantify this clarity using the \textbf{Silhouette Coefficient} ($S$) and demonstrate that (1) high $S$ correlates strongly with RL performance; (2) low $S$ is associated with severe logic errors and reasoning instability. To confirm this property, we introduce a Silhouette-Aware Reweighting strategy that prioritizes low-$S$ samples during training. Experiments across six mathematical benchmarks show consistent improvements across all model families, with gains up to 5.9 points on AIME24. Our work establishes distributional clarity as a fundamental, trainable property underlying RL-Friendliness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。