构建个性化大模型评估基准，让模型更懂用户独特偏好。
PersonalLLM: Tailoring LLMs to Individual Preferences
- 用预训练奖励模型生成多样化用户偏好，模拟真实个性差异。
- 在少量用户反馈下，通过历史数据提升个性化表现。
- 适合研究个性化推荐、用户建模与持续学习的学者使用。
随着大语言模型能力增强，个性化交互潜力日益凸显。我们提出公开基准 PersonalLLM，聚焦于将模型适配至特定用户的细微偏好。不同于以往假设偏好统一的对齐评测，我们构建了开放式提示与多高质量回答组合，体现用户潜在偏好的异质性。不依赖高阶属性（如种族或回复长度）进行角色设定，而是利用一组预训练奖励模型，生成大量具有多样性的虚拟用户。该数据集为应对用户反馈稀疏问题提供了新测试平台——通过借鉴相似用户的历史数据实现个性化优化。我们评估了基础的上下文学习与元学习方法，验证了 PersonalLLM 的价值，并指出了未来算法发展的方向。数据集已发布于 https://huggingface.co/datasets/namkoong-lab/PersonalLLM。
原文摘要 · Abstract (English)
As LLMs become capable of complex tasks, there is growing potential for personalized interactions tailored to the subtle and idiosyncratic preferences of the user. We present a public benchmark, PersonalLLM, focusing on adapting LLMs to provide maximal benefits for a particular user. Departing from existing alignment benchmarks that implicitly assume uniform preferences, we curate open-ended prompts paired with many high-quality answers over which users would be expected to display heterogeneous latent preferences. Instead of persona-prompting LLMs based on high-level attributes (e.g., user's race or response length), which yields homogeneous preferences relative to humans, we develop a method that can simulate a large user base with diverse preferences from a set of pre-trained reward models. Our dataset and generated personalities offer an innovative testbed for developing personalization algorithms that grapple with continual data sparsity--few relevant feedback from the particular user--by leveraging historical data from other (similar) users. We explore basic in-context learning and meta-learning baselines to illustrate the utility of PersonalLLM and highlight the need for future methodological development. Our dataset is available at https://huggingface.co/datasets/namkoong-lab/PersonalLLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。