动态调整数据生成与优化，让大模型更贴合人类偏好。
Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
- 用轻量元学习器实时评估在线数据价值，指导精准采样。
- 在多个评测集上超越现有方法，同时降低42%的标注成本。
- 适合追求高效对齐且资源有限的模型训练团队。
偏好优化对对齐大语言模型与人类价值观至关重要。当前主要挑战在于预收集的离线偏好数据与不断演化的模型策略间存在分布差异。现有方法多依赖静态启发式或解耦的在线采样策略，难以适应模型动态学习状态。为此，我们提出元加权自适应偏好优化（MetaAPO），一种将数据生成与模型训练动态耦合的新框架。MetaAPO采用轻量级元学习器作为“对齐差距估计器”，评估在线策略采样的潜在收益，据此引导针对性的在线数据生成，并为每个样本分配元权重，动态平衡在线与离线数据的质量和分布。在AlpacaEval 2、Arena-Hard和MT-Bench上的实验表明，MetaAPO在多种设置下均持续优于现有方法，同时减少42%的在线标注成本。代码已公开于https://github.com/junming-yang/MetaAPO。
原文摘要 · Abstract (English)
Preference optimization is crucial for aligning large language models (LLMs) with human values and intentions. A significant challenge in this process is the distribution mismatch between pre-collected offline preference data and the evolving model policy. Existing methods attempt to reduce this gap using static heuristics or decoupled online sampling strategies, but they often fail to adapt to the model's dynamic learning state. To bridge this gap, we propose Meta-Weighted Adaptive Preference Optimization (MetaAPO), a novel framework that dynamically couples data generation with model training. MetaAPO employs a lightweight meta-learner, as an "alignment gap estimator", to evaluate the potential benefits of on-policy sampling in relation to offline data. This guides targeted online generation and assigns sample-wise meta-weights to the optimization objective, dynamically balancing the quality and distribution of online and offline data. Experiments on AlpacaEval 2, Arena-Hard and MT-Bench demonstrate that MetaAPO consistently outperforms existing preference optimization approaches across various settings, while reducing 42% in online annotation costs. Code is available at https://github.com/junming-yang/MetaAPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。