让大模型同时满足多种冲突需求，动态调整偏好权重提升对齐效果。
Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
- 用元学习动态生成偏好权重，而非固定分配。
- 在多个基准上优于现有方法，稳定提升多目标对齐性能。
- 适合需要兼顾公平、安全与有用性的大模型训练场景。
多目标对齐旨在通过同时优化多个目标，使大型语言模型（LLMs）符合多样且常冲突的人类价值观。现有方法主要依赖静态偏好权重构建策略，但僵化地对齐固定目标会忽略训练过程中蕴含的有效偏好权衡信息，即使响应偏离目标也可能包含合理权衡。为此，我们提出 Meta-Aligner（MeAL），一种双层元学习框架，实现偏好与策略响应间的双向优化，生成具有指导意义的动态偏好以实现更稳定的训练。具体而言，我们引入偏好权重网络作为元学习器，根据输入提示生成自适应偏好权重，并将其作为可学习参数更新；而 LLM 策略作为基学习器，基于这些偏好使用拒绝采样策略生成响应。大量实验证明，该方法在多个多目标基准上均取得更优表现，验证了动态双向偏好-策略优化框架的有效性。
原文摘要 · Abstract (English)
Multi-Objective Alignment aims to align Large Language Models (LLMs) with diverse and often conflicting human values by optimizing multiple objectives simultaneously. Existing methods predominantly rely on static preference weight construction strategies. However, rigidly aligning to fixed targets discards valuable intermediate information, as training responses inherently embody valid preference trade-offs even when deviating from the target. To address this limitation, we propose Meal, i.e., MEta ALigner, a bi-level meta-learning framework enabling bidirectional optimization between preferences and policy responses, generating instructive dynamic preferences for steadier training. Specifically, we introduce a preference-weight-net as a meta-learner to generate adaptive preference weights based on input prompts and update the preference weights as learnable parameters, while the LLM policy acts as a base-learner optimizing response generation conditioned on these preferences with rejection sampling strategy. Extensive empirical results demonstrate that our method achieves superior performance on several multi-objective benchmarks, validating the effectiveness of the dynamic bidirectional preference-policy optimization framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。