通过协作偏好学习，让大模型更懂用户个性化需求。
CoPL: Collaborative Preference Learning for Personalizing LLMs
- 构建图模型捕捉用户与回复间的关联关系，提升稀疏标注下的偏好估计
- 融合LoRA专家网络，在不重新训练的情况下实现高效微调
- 无需微调即可适应新用户，适合大规模个性化场景
个性化大语言模型对对齐用户多样化偏好至关重要，但现有方法在灵活性和泛化能力上存在不足。本文提出CoPL（协作偏好学习），一种基于图的协同过滤框架，通过建模用户-回复关系来增强偏好估计，尤其适用于标注稀疏的场景。通过整合多组LoRA专家，CoPL能够高效微调大模型，并动态平衡共性与用户特定偏好。此外，无需优化的适配策略使其可直接推广至未见用户，无需微调。在UltraFeedback-P数据集上的实验表明，CoPL优于现有个性化奖励模型，能有效捕捉共通及争议性偏好，是一种可扩展的个性化对齐方案。代码已开源：https://github.com/ml-postech/CoPL。
原文摘要 · Abstract (English)
Personalizing large language models (LLMs) is important for aligning outputs with diverse user preferences, yet existing methods struggle with flexibility and generalization. We propose CoPL (Collaborative Preference Learning), a graph-based collaborative filtering framework that models user-response relationships to enhance preference estimation, particularly in sparse annotation settings. By integrating a mixture of LoRA experts, CoPL efficiently fine-tunes LLMs while dynamically balancing shared and user-specific preferences. Additionally, an optimization-free adaptation strategy enables generalization to unseen users without fine-tuning. Experiments on UltraFeedback-P demonstrate that CoPL outperforms existing personalized reward models, effectively capturing both common and controversial preferences, making it a scalable solution for personalized LLM alignment. The code is available at https://github.com/ml-postech/CoPL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。