用自适应奖励模型让大模型更抗欺骗，少依赖人工设计提示。
Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models
- 元奖励模型动态调整提示，持续优化奖励信号
- 性能媲美甚至超越手工设计提示的模型，且跨任务通用
- 适合追求鲁棒对齐、减少人工干预的研究者与开发者
基于奖励的大语言模型对齐方法面临两大挑战：易受奖励欺骗（reward hacking）影响；当使用大模型作为奖励模型时，严重依赖繁琐的人工提示工程。本文提出元策略优化（MPO），通过引入一个元奖励模型，在训练过程中动态优化奖励模型的提示。该元模型根据训练上下文变化实时调整提示，确保奖励信号始终具有高对齐性，有效抵抗策略模型的投机行为。此元学习机制提升了策略优化的稳定性，显著减少对人工设计奖励提示的依赖。实验表明，MPO在写作、数学推理等多样化任务上均表现优异，无需针对不同任务定制奖励设计。相比标准的RLAIF，MPO具备更强可扩展性，适用于更高阶的对齐框架。整体上，该方法解决了大模型强化学习对齐中的理论与实践难题，为更鲁棒、自适应的对齐策略提供新路径。代码与数据见：https://github.com/minnesotanlp/mpo
原文摘要 · Abstract (English)
Reward-based alignment methods for large language models (LLMs) face two key limitations: vulnerability to reward hacking, where models exploit flaws in the reward signal; and reliance on brittle, labor-intensive prompt engineering when LLMs are used as reward models. We introduce Meta Policy Optimization (MPO), a framework that addresses these challenges by integrating a meta-reward model that dynamically refines the reward model's prompt throughout training. In MPO, the meta-reward model monitors the evolving training context and continuously adjusts the reward model's prompt to maintain high alignment, providing an adaptive reward signal that resists exploitation by the policy. This meta-learning approach promotes a more stable policy optimization, and greatly reduces the need for manual reward prompt design. It yields performance on par with or better than models guided by extensively hand-crafted reward prompts. Furthermore, we show that MPO maintains its effectiveness across diverse tasks, from essay writing to mathematical reasoning, without requiring specialized reward designs. Beyond standard RLAIF, MPO's meta-learning formulation is readily extensible to higher-level alignment frameworks. Overall, this method addresses theoretical and practical challenges in reward-based RL alignment for LLMs, paving the way for more robust and adaptable alignment strategies. The code and data can be accessed at: https://github.com/minnesotanlp/mpo
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。