用博弈方法动态选奖励模型，让大模型训练更准更快。
LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits
- 将奖励模型选择建模为多臂老虎机问题，按需动态选最优模型。
- 在多个任务上提升准确率2.67%以上，速度提升2倍。
- 适合需要高效高精度对齐的大模型应用，如对话和长文本生成。
奖励模型（RMs）对齐大语言模型至关重要，但其在不同任务间的泛化能力难以预知，单一固定模型常导致性能不佳。同时，同时使用多个奖励模型成本高昂且信号冲突。为此，我们提出LASeR（学习自适应选择奖励模型），将奖励模型选择建模为多臂老虎机问题，通过迭代方式为每条输入动态选择最合适的奖励模型进行训练。在常识推理与数学推理任务中，LASeR使Llama-3-8B的平均准确率绝对提升2.67%，相比集成奖励分数表现更优，且效率显著（如提速2倍）。在WildChat（开放式指令遵循任务）中，其AlpacaEval胜率达72.69%，优于基准。在长上下文生成任务中，单文档问答的F1值平均提升2.96点，少样本学习任务提升2.97点，均优于基于最佳n采样的奖励分数集成基线。
原文摘要 · Abstract (English)
Reward Models (RMs) are crucial to aligning large language models (LLMs), but the degree to which an RM specialized to one task (e.g. writing) generalizes to new tasks (e.g. math) is often not known a priori, often making using only one fixed RM to train LLMs suboptimal. However, optimizing LLMs with multiple RMs simultaneously can incur a prohibitively high computational cost and lead to conflicting signals from different RMs that may degrade performance. To address these challenges, we introduce LASeR (Learning to Adaptively Select Rewards), which frames reward model selection as a multi-armed bandit problem, efficiently and iteratively training LLMs using multiple RMs by selecting the most well-suited RM for each instance. On commonsense and math reasoning tasks, we show that LASeR boosts iterative LLM training, improving the absolute average accuracy of Llama-3-8B over three datasets by 2.67% over an ensemble of RM scores while also showing superior efficiency (e.g., a 2x speedup). Moreover, on WildChat (open-ended instruction-following tasks), LASeR leads to a 72.69% AlpacaEval win rate over the RM score ensemble baseline. Extending to long-context generation, LASeR improves by 2.96 F1 points (avg.) on single-document QA tasks and 2.97 F1 points on few-shot learning over the RM score ensemble baseline with best-of-n sampling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。