用检查清单替代奖励模型，让大模型更精准理解用户指令。
Checklists Are Better Than Reward Models For Aligning Language Models
- 从指令中提取检查清单，用AI与专用程序评分生成奖励信号。
- 在5个基准上均提升性能，最高提升6个百分点，硬任务效果显著。
- 适合需要多维度精准响应的场景，如复杂任务规划与安全控制。
语言模型需适应用户指令理解与执行。传统强化学习依赖固定标准（如有用性、有害性），本文提出一种新方法——基于检查清单的强化学习（RLCF）。从指令中提取具体检查项，通过AI评判和专用验证程序评估每项满足程度，整合得分作为强化学习奖励。在强指令跟随模型Qwen2.5-7B-Instruct上测试，五项基准表现全面优于对比方法：FollowBench硬满足率提升4点，InFoBench提升6点，Arena-Hard胜率提升3点。结果表明，检查清单反馈是提升模型应对复杂需求能力的关键工具。
原文摘要 · Abstract (English)
Language models must be adapted to understand and follow user instructions. Reinforcement learning is widely used to facilitate this -- typically using fixed criteria such as "helpfulness" and "harmfulness". In our work, we instead propose using flexible, instruction-specific criteria as a means of broadening the impact that reinforcement learning can have in eliciting instruction following. We propose "Reinforcement Learning from Checklist Feedback" (RLCF). From instructions, we extract checklists and evaluate how well responses satisfy each item - using both AI judges and specialized verifier programs - then combine these scores to compute rewards for RL. We compare RLCF with other alignment methods applied to a strong instruction following model (Qwen2.5-7B-Instruct) on five widely-studied benchmarks -- RLCF is the only method to improve performance on every benchmark, including a 4-point boost in hard satisfaction rate on FollowBench, a 6-point increase on InFoBench, and a 3-point rise in win rate on Arena-Hard. These results establish checklist feedback as a key tool for improving language models' support of queries that express a multitude of needs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。