arXiv:2512.13077cs.LG2025-12被引 1

评测大模型个性化中用户喜欢程度,发现记忆好不等于讨人喜欢。

LikeBench: Evaluating Subjective Likability in LLMs for Personalization

  • 用多轮对话动态评估模型对用户偏好的适应能力
  • 发现记忆准确率高不一定更讨喜,如DeepSeek R1 likability比Qwen3高28%
  • 首次拆解喜好度为7个维度,适合研究个性化与用户体验的学者

个性化大模型应记住用户信息、正确应用并随时间适应偏好。现有基准主要关注信息回忆与应用准确性,但忽视了主观且关键的“喜好度”。为此,我们提出LikeBench——一个基于多轮动态对话的评估框架,通过模拟用户与模型交互,仅从对话中学习偏好,并在每轮后从七个维度评估模型回应的喜好度:情感适应、正式程度匹配、知识适应、引用理解、对话长度适配、幽默契合度和回调能力。该框架采用精细的心理学人格描述角色,而非粗粒度评分,使评估更真实。实验表明,记忆表现强并不意味着喜好度高:尽管深思(DeepSeek R1)记忆准确率较低(86%,每用户17条事实),其喜好度得分仍比通义千问3(Qwen3,93%准确率,43条事实)高出28%;即使是顶尖模型如GPT-5,在短对话中表现良好,但在长而嘈杂的交互中适应性有限。

原文摘要 · Abstract (English)

A personalized LLM should remember user facts, apply them correctly, and adapt over time to provide responses that the user prefers. Existing LLM personalization benchmarks are largely centered on two axes: accurately recalling user information and accurately applying remembered information in downstream tasks. We argue that a third axis, likability, is both subjective and central to user experience, yet under-measured by current benchmarks. To measure likability holistically, we introduce LikeBench, a multi-session, dynamic evaluation framework that measures likability across multiple dimensions by how much an LLM can adapt over time to a user's preferences to provide more likable responses. In LikeBench, the LLMs engage in conversation with a simulated user and learn preferences only from the ongoing dialogue. As the interaction unfolds, models try to adapt to responses, and after each turn, they are evaluated for likability across seven dimensions by the same simulated user. To the best of our knowledge, we are the first to decompose likability into multiple diagnostic metrics: emotional adaptation, formality matching, knowledge adaptation, reference understanding, conversation length fit, humor fit, and callback, which makes it easier to pinpoint where a model falls short. To make the simulated user more realistic and discriminative, LikeBench uses fine-grained, psychologically grounded descriptive personas rather than the coarse high/low trait rating based personas used in prior work. Our benchmark shows that strong memory performance does not guarantee high likability: DeepSeek R1, with lower memory accuracy (86%, 17 facts/profile), outperformed Qwen3 by 28% on likability score despite Qwen3's higher memory accuracy (93%, 43 facts/profile). Even SOTA models like GPT-5 adapt well in short exchanges but show only limited robustness in longer, noisier interactions.

大模型评估个性化用户偏好对话系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。