用大模型理解自由文本推荐请求,效果远超传统方法。
Large Language Models as Narrative-Driven Recommenders
- 用38个大小不一的LLM处理用户自由描述的电影需求
- 大模型在真实数据集上显著优于doc2vec等传统方法
- 零样本提示已足够有效,适合快速落地应用
叙事驱动推荐系统旨在根据用户以自然语言表达的需求(如“我想看一部像《禁闭岛》那样烧脑的悬疑片”)提供个性化推荐。尽管大语言模型(LLMs)在处理通用自然语言方面表现出色,但其在处理此类推荐请求方面的有效性仍研究不足。为此,我们评估了38个不同规模的开源与闭源LLM(如LLama 3.2和GPT-4o)在电影推荐场景中的表现。实验基于Reddit电影推荐社区的高质量人工标注数据集,并采用零样本、身份提示和少样本提示等多种策略。结果表明,LLMs能生成上下文相关性强的电影推荐,显著优于现有先进方法(如doc2vec)。虽然闭源大参数模型整体表现最佳,但中等规模的开源模型也表现良好,仅略逊于计算成本更高的模型。此外,多数模型在不同提示策略间无显著差异,说明零样本提示对叙事驱动推荐已足够有效。本研究为推荐系统研究人员及希望将LLM集成到实际推荐工具中的从业者提供了重要参考。
原文摘要 · Abstract (English)
Narrative-driven recommenders aim to provide personalized suggestions for user requests expressed in free-form text such as "I want to watch a thriller with a mind-bending story, like Shutter Island." Although large language models (LLMs) have been shown to excel in processing general natural language queries, their effectiveness for handling such recommendation requests remains relatively unexplored. To close this gap, we compare the performance of 38 open- and closed-source LLMs of various sizes, such as LLama 3.2 and GPT-4o, in a movie recommendation setting. For this, we utilize a gold-standard, crowdworker-annotated dataset of posts from reddit's movie suggestion community and employ various prompting strategies, including zero-shot, identity, and few-shot prompting. Our findings demonstrate the ability of LLMs to generate contextually relevant movie recommendations, significantly outperforming other state-of-the-art approaches, such as doc2vec. While we find that closed-source and large-parameterized models generally perform best, medium-sized open-source models remain competitive, being only slightly outperformed by their more computationally expensive counterparts. Furthermore, we observe no significant differences across prompting strategies for most models, underscoring the effectiveness of simple approaches such as zero-shot prompting for narrative-driven recommendations. Overall, this work offers valuable insights for recommender system researchers as well as practitioners aiming to integrate LLMs into real-world recommendation tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。