混合奖励机制让大模型推理更准，兼顾正确性与细节优化
Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense
- 用分层归一化融合验证器与奖励模型信号，保持正确性同时细化评分
- 在多个数学推理任务上优于纯验证或纯奖励模型方法，尤其擅长难验证题
- 适合需要高精度推理的场景，如数学竞赛、代码生成等复杂任务
大语言模型的推理能力提升越来越多依赖可验证的奖励：提供0-1正确性信号的确定性检查器。尽管可靠，这种二元反馈过于僵硬——许多任务存在部分正确或替代答案，验证器难以充分赋分，全有或全无的监督限制了学习效果。奖励模型则能提供更丰富的连续反馈，可作为验证器的补充信号。本文提出HERO（混合集成奖励优化）框架，以结构化方式整合验证器信号与奖励模型得分。HERO采用分层归一化，将奖励模型得分限定在验证器定义的组内，既保留正确性又细化质量差异；并使用方差感知加权，在挑战性提示中突出密集信号的作用。在多种数学推理基准测试中，HERO持续优于仅用奖励模型或仅用验证器的基线，尤其在可验证和难以验证的任务上表现显著。结果表明，混合奖励设计在保持验证器稳定性的同时，利用奖励模型的细微差异推动推理能力提升。
原文摘要 · Abstract (English)
Post-training for reasoning of large language models (LLMs) increasingly relies on verifiable rewards: deterministic checkers that provide 0-1 correctness signals. While reliable, such binary feedback is brittle--many tasks admit partially correct or alternative answers that verifiers under-credit, and the resulting all-or-nothing supervision limits learning. Reward models offer richer, continuous feedback, which can serve as a complementary supervisory signal to verifiers. We introduce HERO (Hybrid Ensemble Reward Optimization), a reinforcement learning framework that integrates verifier signals with reward-model scores in a structured way. HERO employs stratified normalization to bound reward-model scores within verifier-defined groups, preserving correctness while refining quality distinctions, and variance-aware weighting to emphasize challenging prompts where dense signals matter most. Across diverse mathematical reasoning benchmarks, HERO consistently outperforms RM-only and verifier-only baselines, with strong gains on both verifiable and hard-to-verify tasks. Our results show that hybrid reward design retains the stability of verifiers while leveraging the nuance of reward models to advance reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。