大模型推荐能力受评分偏差干扰，新评测基准揭示其偏好捕捉短板
Can Large Language Models Understand Preferences in Personalized Recommendation?
- 构建分组排序评测框架，剥离评分偏差与物品质量影响
- 大模型在消除干扰后仍难识别用户真正偏好，尤其小模型表现差
- 推荐效果与传统评分误差相关性低，适合研究个性化推荐的学者
大型语言模型在个性化推荐中表现优异，但现有评估方法多依赖评分预测的回归误差。然而，用户评分偏差和物品质量会掩盖真实偏好。为此，我们提出PerRecBench，通过分组排名方式剥离这两类因素，评估推荐技术对个人偏好的捕捉能力。使用19个LLM和PerRecBench发现，尽管大模型普遍优于小模型，但在消除评分偏差与物品质量后，仍难以识别用户的偏好项与排斥项。结果表明，成对与列表级排序优于点级排序；PerRecBench与传统回归指标相关性低；用户画像与预训练数据分布至关重要。进一步探索三种监督微调策略，发现单格式训练权重合并具潜力，但提升模型理解用户偏好的能力仍是开放问题。代码与数据见https://github.com/TamSiuhin/PerRecBench。
原文摘要 · Abstract (English)
Large Language Models (LLMs) excel in various tasks, including personalized recommendations. Existing evaluation methods often focus on rating prediction, relying on regression errors between actual and predicted ratings. However, user rating bias and item quality, two influential factors behind rating scores, can obscure personal preferences in user-item pair data. To address this, we introduce PerRecBench, disassociating the evaluation from these two factors and assessing recommendation techniques on capturing the personal preferences in a grouped ranking manner. We find that the LLM-based recommendation techniques that are generally good at rating prediction fail to identify users' favored and disfavored items when the user rating bias and item quality are eliminated by grouping users. With PerRecBench and 19 LLMs, we find that while larger models generally outperform smaller ones, they still struggle with personalized recommendation. Our findings reveal the superiority of pairwise and listwise ranking approaches over pointwise ranking, PerRecBench's low correlation with traditional regression metrics, the importance of user profiles, and the role of pretraining data distributions. We further explore three supervised fine-tuning strategies, finding that merging weights from single-format training is promising but improving LLMs' understanding of user preferences remains an open research problem. Code and data are available at https://github.com/TamSiuhin/PerRecBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。