让大模型同时满足有用、安全、简洁等多重人类偏好。
Tangent Space Fine-Tuning for Directional Preference Alignment in Large Language Models
- 在模型切空间中学习各偏好的更新方向，实现可控调节。
- 在帮助性与冗长性权衡上覆盖更广的帕累托最优解集。
- 推理时线性组合方向即可生成指定行为，无需重新训练。
本文旨在通过有原则且可控制的方式，使大语言模型在多个用户偏好维度（如帮助性、安全性、冗长性）间取得平衡。现有偏好优化方法（如直接偏好优化，DPO）将反馈压缩为单一标量奖励，固定了目标间的权衡关系，无法遍历帕累托前沿。受Ortiz-Jimenez等（2023）启发，本文将微调视为模型切空间中的线性更新，提出切空间直接偏好优化（TS-DPO），在局部线性区域执行DPO，学习每个偏好的更新方向。这些方向可在推理时线性组合，生成用户指定行为，无需额外优化。在HelpSteer和UltraFeedback数据集上的实验表明，相较于标量化的DPO，TS-DPO实现了更广的帕累托最优覆盖与更平滑的偏好控制。典型相关分析（CCA）进一步显示，切空间训练增强了与不同偏好对齐的主成分方向，提升了偏好的解耦性。
原文摘要 · Abstract (English)
Our goal is to enable large language models (LLMs) to balance multiple human preference dimensions; such as helpfulness, safety, and verbosity, through principled and controllable alignment. Existing preference optimization methods, including Direct Preference Optimization (DPO), collapse feedback into a single scalar reward, fixing one balance among objectives and preventing traversal of the Pareto front. Recent work by Ortiz-Jimenez et al. (2023) showed that fine-tuning can be viewed in a model's tangent space, where linearized updates act as additive vectors that can be composed to jointly perform well on multiple tasks. Building on this formulation, we extend this idea to preference alignment and propose Tangent-Space Direct Preference Optimization (TS-DPO), which performs DPO within this locally linear regime to learn per-objective update directions. These directions can be linearly combined at inference to generate user-specified behaviors without additional optimization. Evaluated on the helpfulness-verbosity trade-off using the HelpSteer and UltraFeedback datasets, TS-DPO achieves broader Pareto-optimal coverage and smoother preference control than scalarized DPO. Canonical Correlation Analysis (CCA) further shows that tangent-space training amplifies canonical directions aligned with distinct preferences, improving disentanglement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。