用少量用户反馈实现大模型个性化，让回复更贴合个人偏好。
Language Model Personalization via Reward Factorization
- 将用户偏好建模为基础奖励函数的线性组合，实现高效个性化。
- 仅需约10条用户反馈，就能显著提升模型输出与用户偏好的匹配度。
- 适合需要个性化对话系统的场景，如智能助手、教育辅导。
现代大型语言模型（LLMs）通过人类反馈强化学习（RLHF）优化以生成符合人类价值观的回应。然而现有方法假设存在统一的偏好模型，未考虑个体用户差异，限制了其在个性化应用中的效果。本文提出一种框架，通过假设用户偏好位于低维空间，将用户特定奖励表示为基底奖励函数的线性组合，从而扩展RLHF实现用户个性化。无需为每位用户训练独立模型，仅需约10条用户反馈即可推断出用户特定奖励，并据此对齐模型输出。我们在合成数据和真实用户上进行实验，验证了该方法的有效性。人机评估显示，该方法在67%的对比中胜过默认GPT-4o的响应。
原文摘要 · Abstract (English)
Modern large language models (LLMs) are optimized for human-aligned responses using Reinforcement Learning from Human Feedback (RLHF). However, existing RLHF approaches assume a universal preference model and fail to account for individual user preferences, limiting their effectiveness in personalized applications. We introduce a framework that extends RLHF to enable user personalization by leveraging the assumption that user preferences lie in a low-dimensional space. Instead of training a separate model per user, we represent user-specific rewards as a linear combination of base reward functions. Using only ~10 user responses, our method can infer user-specific rewards and align LLM outputs accordingly. We validate our approach through experiments with both synthetic and real users, demonstrating significant personalization achieved by our method. In human evaluations, our method achieves a 67% win rate over default GPT-4o responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。