提出新数据集与评估方法,让推荐解释更真实反映用户喜恶。
Disentangling Likes and Dislikes in Personalized Generative Explainable Recommendation
- 用大模型从用户评论中提取正负情感,构建新数据集
- 现有模型在传统指标上表现好,但情感对齐度差
- 输入用户预测评分可提升解释的情感感知能力
现有可解释推荐研究多将任务视为标准文本生成,仅通过生成解释与真实解释的文本相似度评估模型。然而,这忽略了系统关键问题:生成内容是否准确反映用户(购买后)的真实情感,即为何喜欢或不喜欢推荐物品。为此,我们构建了新数据集,通过大模型显式提取用户评论中的正负意见,并提出新评估方法:衡量生成解释是否与用户情感一致,以及能否准确识别用户对目标商品的正负观点。我们在新数据集上测试多个近期模型,发现高传统指标表现并不保证情感对齐。最后发现,若将用户对目标商品的预测评分直接作为输入,现有模型能生成更具情感感知的解释。数据集与基准实现已开源。
原文摘要 · Abstract (English)
Recent research on explainable recommendation generally frames the task as a standard text generation problem, and evaluates models simply based on the textual similarity between the predicted and ground-truth explanations. However, this approach fails to consider one crucial aspect of the systems: whether their outputs accurately reflect the users' (post-purchase) sentiments, i.e., whether and why they would like and/or dislike the recommended items. To shed light on this issue, we introduce new datasets and evaluation methods that focus on the users' sentiments. Specifically, we construct the datasets by explicitly extracting users' positive and negative opinions from their post-purchase reviews using an LLM, and propose to evaluate systems based on whether the generated explanations 1) align well with the users' sentiments, and 2) accurately identify both positive and negative opinions of users on the target items. We benchmark several recent models on our datasets and demonstrate that achieving strong performance on existing metrics does not ensure that the generated explanations align well with the users' sentiments. Lastly, we find that existing models can provide more sentiment-aware explanations when the users' (predicted) ratings for the target items are directly fed into the models as input. The datasets and benchmark implementation are available at: https://github.com/jchanxtarov/sent_xrec.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。