通过对齐正确推理路径的隐藏状态，提升大模型数学推理能力。
Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning
- 在答案前一词位置对齐正确推理的隐藏状态，增强统一决策表征。
- 在8个数学推理基准上，性能相比DAPO基线提升3.8至6.2个百分点。
- 无需额外计算开销，适用于不同规模的模型，可推广至各类强化学习推理任务。
基于可验证奖励的强化学习（RLVR）已成为提升大语言模型数学推理能力的主流方法，但现有方法将每个正确推理轨迹简化为单一奖励比特，忽略了其隐藏状态间共享的几何结构。我们发现，在锚点位置（即答案标记前一个词），正确轨迹因需输出相同答案而自然收敛（余弦相似度约0.84），但仍保留各自推理路径带来的残余差异。促使该位置完全对齐，可使模型提取统一的“正确决策”表征，降低对具体推理路径的敏感性。基于此，我们提出Hidden-Align——一种在强化学习训练中于锚点位置对齐正确轨迹最后一层隐藏状态的辅助损失函数，训练与推理开销为零。在八个数学推理基准上，Hidden-Align相较于DAPO基线，在Qwen3-1.7B、4B、14B模型上平均通过率（pass@1）分别提升3.8、6.2、5.4个百分点，且在所有规模下均实现一致的pass@k提升，消融实验验证了损失类型、锚点位置、层深及权重的影响。
原文摘要 · Abstract (English)
Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states. Investigating this structure, we find that at the anchor token (the position immediately before the answer marker), correct rollouts converge naturally because they must produce the same answer (cosine similarity ~0.84), yet each retains residual variance from its unique reasoning path. Encouraging full alignment at this point pushes the model to extract a unified "correct decision" representation, reducing sensitivity to which reasoning path was taken. Based on this observation, we propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference. On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales, supported by ablations on loss type, anchor position, layer depth, and loss weight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。