用低秩建模让大模型快速适配用户偏好，少样本也能精准预测。
LoRe: Personalizing LLMs via Low-Rank Reward Modeling
- 用低秩子空间表示奖励函数，共享基函数组合出个体偏好
- 在多个数据集上对未见用户泛化效果更好，预测准确率更高
- 适合需要快速个性化且数据有限的场景，如智能助手
将大语言模型个性化以适应多样用户偏好，对提升对齐度和用户满意度至关重要。传统基于人类反馈的强化学习（RLHF）方法通常依赖单一价值表示，难以适应个体差异。本文提出一种新框架，通过低秩偏好建模高效学习并泛化用户专属奖励函数。将奖励函数表示在低维子空间中，并将个体偏好建模为共享基函数的加权组合，避免了僵化的用户分类，同时实现可扩展性和少样本适应。我们在多个偏好数据集上验证该方法，结果表明其对未见用户的泛化能力更优，且在偏好预测任务中准确率更高。
原文摘要 · Abstract (English)
Personalizing large language models (LLMs) to accommodate diverse user preferences is essential for enhancing alignment and user satisfaction. Traditional reinforcement learning from human feedback (RLHF) approaches often rely on monolithic value representations, limiting their ability to adapt to individual preferences. We introduce a novel framework that leverages low-rank preference modeling to efficiently learn and generalize user-specific reward functions. By representing reward functions in a low-dimensional subspace and modeling individual preferences as weighted combinations of shared basis functions, our approach avoids rigid user categorization while enabling scalability and few-shot adaptation. We validate our method on multiple preference datasets, demonstrating superior generalization to unseen users and improved accuracy in preference prediction tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。