评测大模型动态用户画像与个性化回应能力，发现现有模型表现仅约50%准确。
Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale
- 构建包含180个模拟用户的对话历史，覆盖15项任务共60轮多轮交互。
- 前沿模型如GPT-4.5、Gemini-2.0在动态用户偏好识别上准确率仅约50%。
- 适合研究个性化对话系统、用户建模与长期记忆机制的学者使用。
大语言模型（LLMs）已成为各类任务中的个性化助手，从写作支持到定制推荐和咨询。用户与LLM的交互历史可反映其个性特征与偏好。然而，当前仍不清楚模型如何有效利用这些历史来（1）内化用户的固有特质与偏好，（2）追踪偏好随时间的变化，（3）在新场景中生成个性化响应。本文提出PERSONAMEM基准，包含超过180个模拟用户的交互历史，每个用户有最多60轮跨15项真实任务的多轮对话。针对用户以第一人称提出的即时查询，评估模型根据用户当前画像生成合适响应的能力。结果发现，当前模型通过直接提示难以识别用户画像的动态演变，导致响应与用户当前情境和偏好不匹配，即使前沿模型如GPT-4.1、o4-mini、GPT-4.5、o1或Gemini-2.0整体准确率也仅约50%，仍有提升空间。我们希望PERSONAMEM及其用户画像与对话模拟流程能推动真正用户感知型聊天机器人的发展。代码与数据见github.com/bowen-upenn/PersonaMem。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have emerged as personalized assistants for users across a wide range of tasks -- from offering writing support to delivering tailored recommendations or consultations. Over time, the interaction history between a user and an LLM can provide extensive information about an individual's traits and preferences. However, open questions remain on how well LLMs today can effectively leverage such history to (1) internalize the user's inherent traits and preferences, (2) track how the user profiling and preferences evolve over time, and (3) generate personalized responses accordingly in new scenarios. In this work, we introduce the PERSONAMEM benchmark. PERSONAMEM features curated user profiles with over 180 simulated user-LLM interaction histories, each containing up to 60 sessions of multi-turn conversations across 15 real-world tasks that require personalization. Given an in-situ user query, i.e. query issued by the user from the first-person perspective, we evaluate LLM chatbots' ability to identify the most suitable response according to the current state of the user's profile. We observe that current LLMs still struggle to recognize the dynamic evolution in users' profiles over time through direct prompting approaches. As a consequence, LLMs often fail to deliver responses that align with users' current situations and preferences, with frontier models such as GPT-4.1, o4-mini, GPT-4.5, o1, or Gemini-2.0 achieving only around 50% overall accuracy, suggesting room for improvement. We hope that PERSONAMEM, along with the user profile and conversation simulation pipeline, can facilitate future research in the development of truly user-aware chatbots. Code and data are available at github.com/bowen-upenn/PersonaMem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。