用用户互动生成虚拟人格，让大模型更懂个人偏好。
SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs
- 从用户对话中推理出偏好原因，生成虚拟人格
- 提升个性化评分模型准确率4.4%（Chatbot Arena）
- 适合需要精准个性化对齐的研究与应用
近期关于大语言模型多元对齐的呼声推动了模型对用户偏好的适配。然而，现有个性化评分模型大多依赖额外身份信息，如人口统计或预设偏好类别。为此，我们提出SynthesizeMe，一种从用户交互中生成合成用户人格的方法。该方法首先生成并验证解释用户偏好的推理过程，再基于此推理推导出合成用户人格，并筛选出对特定用户有信息量的前置交互，构建个性化提示。实验表明，使用SynthesizeMe生成的提示使个性化大模型作为评判者在Chatbot Arena上的准确率提升4.4%。结合合成提示与评分模型，在PersonalRewardBench上取得最佳表现——该数据集由854名Chatbot Arena用户与聊天机器人交互的分层数据构成，源自PRISM。
原文摘要 · Abstract (English)
Recent calls for pluralistic alignment of Large Language Models (LLMs) encourage adapting models to diverse user preferences. However, most prior work on personalized reward models heavily rely on additional identity information, such as demographic details or a predefined set of preference categories. To this end, we introduce SynthesizeMe, an approach to inducing synthetic user personas from user interactions for personalized reward modeling. SynthesizeMe first generates and verifies reasoning to explain user preferences, then induces synthetic user personas from that reasoning, and finally filters to informative prior user interactions in order to build personalized prompts for a particular user. We show that using SynthesizeMe induced prompts improves personalized LLM-as-a-judge accuracy by 4.4% on Chatbot Arena. Combining SynthesizeMe derived prompts with a reward model achieves top performance on PersonalRewardBench: a new curation of user-stratified interactions with chatbots collected from 854 users of Chatbot Arena and PRISM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。