用符号化工具让大模型更精准理解人类多维偏好。
UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality
- 用非线性效用函数将偏好转为符号令牌，避免数值敏感问题。
- 仅训练一个模型即实现更优的帕累托前沿，计算开销更低。
- 适合需要高精度对齐多目标人类偏好的模型优化场景。
基于人类反馈的强化学习（RLHF）已成为对齐大语言模型与人类价值观的核心方法。然而，现有方法难以捕捉人类偏好的多维分布特性。直接将原始奖励值注入提示的RiC方法存在显著数值敏感性问题——例如模型可能无法区分9.11与9.8；而MORLHF、Rewarded Soups和MODPO等替代方案则需训练多个模型，带来高昂计算成本。本文提出一种新框架：效用条件多目标对齐（UC-MOA）。该方法利用一组严格递增的非线性效用函数，将用户指定偏好转化为符号令牌，并以此条件化单一语言模型。该设计既缓解了数值推理挑战，又大幅降低训练开销，使模型在复杂奖励维度上实现更优的帕累托前沿与更强的对齐能力。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone for aligning large language models (LLMs) with human values. However, existing approaches struggle to capture the multi-dimensional, distributional nuances of human preferences. Methods such as RiC that directly inject raw reward values into prompts face significant numerical sensitivity issues--for instance, LLMs may fail to distinguish between 9.11 and 9.8--while alternatives like MORLHF, Rewarded Soups, and MODPO incur high computational costs by training multiple models. In this work, we introduce Utility-Conditioned Multi-Objective Alignment (UC-MOA), a novel framework that overcomes these limitations. Our approach leverages a diverse set of strictly increasing, non-linear utility functions to transform user-specified preferences into symbolic tokens, which are then used to condition a single LLM. This design not only mitigates numerical reasoning challenges but also substantially reduces training overhead, yielding models that achieve superior Pareto fronts and robust alignment across complex reward dimensions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。