用贝叶斯方法动态选奖励模型，提升大模型对齐效果
Reward Model Routing in Alignment
- 先离线学各奖励模型优劣，再在线用贝叶斯选最优模型
- 在多个评测集上超越单个模型和现有路由方法
- 适合追求高对齐质量的LLM训练场景
基于人类或人工智能反馈的强化学习（RLHF/RLAIF）已成为对齐大型语言模型（LLMs）的标准范式。然而，多数流程依赖单一奖励模型（RM），限制了对齐质量并存在过拟合风险。近期研究探索奖励模型路由——从候选池中动态选择一个RM，以利用互补优势，同时保持每次查询仅调用一次RM。但现有方法存在冷启动问题且探索不足。本文提出BayesianRouter，一种结合离线学习与在线贝叶斯选择的混合路由框架。离线阶段，通过偏好数据训练多任务路由器，估计各RM的可靠性；在线阶段，采用贝叶斯汤普森采样路由器，以离线嵌入作为高斯先验初始化各RM权重向量，并根据在线奖励自适应更新后验分布，以适应策略分布的变化。在指令遵循（AlpacaEval-2、Arena-Hard、MT-Bench）和推理任务（GSM8K、MMLU）上的大量实验表明，BayesianRouter持续优于单个RM、RM集成及现有路由方法。
原文摘要 · Abstract (English)
Reinforcement learning from human or AI feedback (RLHF / RLAIF) has become the standard paradigm for aligning large language models (LLMs). However, most pipelines rely on a single reward model (RM), limiting alignment quality and risking overfitting. Recent work explores RM routing--dynamically selecting an RM from a candidate pool to exploit complementary strengths while maintaining $O(1)$ RM calls--but existing methods suffer from cold-start and insufficient exploration. We propose BayesianRouter, a hybrid routing framework that combines offline RM strengths learning with online Bayesian selection. In the offline stage, a multi-task router is trained on preference data to estimate per-RM reliability. In the online stage, a Bayesian Thompson sampling router performs per-query RM selection, initializing RM-specific weight vectors with offline embeddings as Gaussian priors and adaptively updating their posteriors with online rewards to adapt to the evolving policy distribution. Extensive experiments on instruction-following (AlpacaEval-2, Arena-Hard, MT-Bench) and reasoning (GSM8K, MMLU) benchmarks show that BayesianRouter consistently outperforms individual RMs, RM ensembling, and existing routing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。