用元学习动态调整奖励，让AI更懂人类偏好且训练更稳定
Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback
- 先在多个任务上元学习奖励形状函数，再用于具体对齐
- 在四个基准测试中胜过传统方法，长文本生成胜率90.8%
- 适合需要高精度对齐与稳定训练的LLM优化场景
从人类反馈中进行强化学习（RLHF）是使大语言模型与人类偏好对齐的标准方法，但其效果受限于静态、任务无关的奖励模型。这种不匹配导致学习信号稀疏，对齐效果不佳。我们提出MeRLa（元学习奖励塑形），一个原则性框架，在正式的RLHF训练前，通过辅助任务元学习一个任务感知的塑形函数Φ(x,y;ϕ)。所学塑形生成复合奖励，在保持策略最优性的同时提供任务特定的学习信号。我们的元目标结合了任务区分性、熵正则化和基于势能的保全性，以实现稳定收敛。我们提供了策略不变性的理论保证，分析了表示漂移敏感性，并正式解决了熵最大化带来的激励错位问题。在LLaMA-3-8B上的实验表明，跨四个基准测试，MeRLa持续优于PPO、DPO、GRPO和DAPO，于AlpacaEval 2.0上达到90.8%的长度控制胜率，MT-Bench得分为9.14，训练不稳定性降低41%。当与基于过程和评分标准的增强奖励结合时，其优势依然存在。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback (RLHF) is the standard approach for aligning large language models with human preferences, but its quality is limited by static, task-agnostic reward models. This mismatch leads to sparse learning signals and suboptimal alignment. We introduce MeRLa (Meta-Learned Reward Shaping), a principled framework that meta-learns a task-aware shaping function $Φ(x,y;ϕ)$ across auxiliary tasks before RLHF training. The learned shaping produces a composite reward that preserves policy optimality while providing task-specific learning signals. Our meta-objective combines task discrimination, entropy regularization, and potential-based conservation for stable convergence. We provide theoretical guarantees for policy invariance, analyze representation drift sensitivity, and formally address incentive misalignment from entropy maximization. Experiments on LLaMA-3-8B across four benchmarks show consistent improvements over PPO, DPO, GRPO, and DAPO, achieving a 90.8% length-controlled win rate on AlpacaEval 2.0 and a score of 9.14 on MT-Bench, with 41% less training instability. MeRLa retains its benefits when combined with process-based and rubric-based enhanced rewards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。