用不确定性路由让快模型和强模型分工，提升对齐效率
Ask a Strong LLM Judge when Your Reward Model is Uncertain
- 根据置信度决定由快模型或强模型判断偏好
- 相同成本下比随机调用强模型效果提升显著
- 适合需要高效在线对齐的强化学习场景
奖励模型（RM）在基于人类反馈的强化学习（RLHF）中对齐大语言模型（LLM）起关键作用。然而，传统基于人类偏好训练的RM易受奖励黑客攻击，且对分布外（OOD）输入泛化能力差。相比之下，具备推理能力的强语言模型判官虽无需额外训练即可实现更好泛化，但推理开销大，难以用于在线RLHF。本文提出一种基于不确定性的路由框架，以高效互补快速的RM与昂贵的强语言模型判官。该方法将策略梯度中的优势估计转化为成对偏好分类，实现可解释的不确定性量化以指导路由：置信度低的样本交由强模型判官处理，高置信度则由RM评估。在多个RM基准上的实验表明，该不确定性路由策略在相同成本下显著优于随机调用判官，下游对齐结果也验证了其在提升在线RLHF方面的有效性。
原文摘要 · Abstract (English)
Reward model (RM) plays a pivotal role in reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs). However, classical RMs trained on human preferences are vulnerable to reward hacking and generalize poorly to out-of-distribution (OOD) inputs. By contrast, strong LLM judges equipped with reasoning capabilities demonstrate superior generalization, even without additional training, but incur significantly higher inference costs, limiting their applicability in online RLHF. In this work, we propose an uncertainty-based routing framework that efficiently complements a fast RM with a strong but costly LLM judge. Our approach formulates advantage estimation in policy gradient (PG) methods as pairwise preference classification, enabling principled uncertainty quantification to guide routing. Uncertain pairs are forwarded to the LLM judge, while confident ones are evaluated by the RM. Experiments on RM benchmarks demonstrate that our uncertainty-based routing strategy significantly outperforms random judge calling at the same cost, and downstream alignment results showcase its effectiveness in improving online RLHF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。