用用户评论提升大模型评分预测能力，效果媲美传统方法。
Enhancing Rating Prediction with Off-the-Shelf LLMs Using In-Context User Reviews
- 用用户评论作为上下文，让现成大模型更精准预测评分。
- 评论越具体（针对具体物品），预测效果越好，最高提升12.3%。
- 先让模型生成假设评论，再预测评分，性能更优，适合冷启动场景。
将大语言模型（LLMs）个性化以匹配个体用户偏好是当前研究热点。然而，以往工作主要关注分类或排序任务，未充分探索需要语言与数学推理结合的李克特量表评分预测（回归任务）。该任务在工业界有广泛应用，但大模型的使用仍不充分，尤其对现成大模型的能力研究不足。本研究在三个数据集上测试了八种现成大模型的评分预测表现，通过不同上下文信息进行对比。实验表明，用户撰写的具体评论显著提升了大模型的评分预测性能，效果可媲美矩阵分解等传统方法，展现出解决冷启动问题的巨大潜力。我们还发现，针对具体物品的评论比泛化偏好描述更有效；此外，引导模型先生成假设评论，能进一步提升预测表现。代码已开源：https://github.com/ynklab/rating-prediction-with-reviews。
原文摘要 · Abstract (English)
Personalizing the outputs of large language models (LLMs) to align with individual user preferences is an active research area. However, previous studies have mainly focused on classification or ranking tasks and have not considered Likert-scale rating prediction, a regression task that requires both language and mathematical reasoning to be solved effectively. This task has significant industrial applications, but the utilization of LLMs remains underexplored, particularly regarding the capabilities of off-the-shelf LLMs. This study investigates the performance of off-the-shelf LLMs on rating prediction, providing different in-context information. Through comprehensive experiments with eight models across three datasets, we demonstrate that user-written reviews significantly improve the rating prediction performance of LLMs. This result is comparable to traditional methods like matrix factorization, highlighting the potential of LLMs as a promising solution for the cold-start problem. We also find that the reviews for concrete items are more effective than general preference descriptions that are not based on any specific item. Furthermore, we discover that prompting LLMs to first generate a hypothetical review enhances the rating prediction performance. Our code is available at https://github.com/ynklab/rating-prediction-with-reviews.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。