用主成分分析从二元比较中拆解出多样人类偏好，实现可解释的个性化对齐。
Rethinking Diverse Human Preference Learning through Principal Component Analysis
- 将偏好表示为向量，通过PCA提取正交偏好维度
- 无需细粒度标注即可识别如帮助性、安全性等关键偏好维度
- 支持新用户快速适配，适合个性化AI系统研发
理解人类偏好对于改进基础模型和构建个性化AI系统至关重要。然而，偏好本身具有多样性和复杂性，传统奖励模型难以全面捕捉。尽管细粒度偏好数据有助于建模，但其收集成本高且难扩展。本文提出分解式奖励模型（DRMs），一种新方法：仅使用二元比较数据，无需细粒度标注，即可从中提取多样人类偏好。核心思路是将人类偏好表示为向量，并利用主成分分析（PCA）进行分析。通过构建优选与劣选响应嵌入差异的数据集，DRMs识别出正交基向量，捕捉不同偏好维度。这些分解后的奖励可灵活组合，以匹配不同用户需求，提供可解释且可扩展的替代方案。实验表明，DRMs能有效提取有意义的偏好维度（如帮助性、安全性、幽默感），并在无额外训练的情况下适应新用户。结果凸显其在个性化与可解释大模型对齐中的强大潜力。代码已开源：https://github.com/amandaluof/DRMs。
原文摘要 · Abstract (English)
Understanding human preferences is crucial for improving foundation models and building personalized AI systems. However, preferences are inherently diverse and complex, making it difficult for traditional reward models to capture their full range. While fine-grained preference data can help, collecting it is expensive and hard to scale. In this paper, we introduce Decomposed Reward Models (DRMs), a novel approach that extracts diverse human preferences from binary comparisons without requiring fine-grained annotations. Our key insight is to represent human preferences as vectors and analyze them using Principal Component Analysis (PCA). By constructing a dataset of embedding differences between preferred and rejected responses, DRMs identify orthogonal basis vectors that capture distinct aspects of preference. These decomposed rewards can be flexibly combined to align with different user needs, offering an interpretable and scalable alternative to traditional reward models. We demonstrate that DRMs effectively extract meaningful preference dimensions (e.g., helpfulness, safety, humor) and adapt to new users without additional training. Our results highlight DRMs as a powerful framework for personalized and interpretable LLM alignment. Our code is available at https://github.com/amandaluof/DRMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。