用变异测试评估ChatGPT推荐系统，发现传统指标不足。
Metamorphic Evaluation of ChatGPT as a Recommender System
- 设计推荐与语言模型双重变异关系，检测输入输出一致性。
- 在MovieLens上测试GPT3.5，Kendall τ与RBO相似度显著降低。
- 适合关注大模型推荐系统评估的学者与工程师。
随着大型语言模型（如ChatGPT）的兴起，研究者尝试将其用于提升推荐效果。然而，这些模型具有黑箱与概率性特征，其评估方式却仍沿用传统推荐系统的指标。为填补这一空白，本文提出针对基于GPT的推荐系统（RS）的变异测试方法。该方法通过定义输入与输出间的变异关系（MRs），并验证输出是否满足这些关系。我们从推荐系统和语言模型两个角度构建了多种变异关系，包括评分乘法/平移（来自推荐系统）以及在提示中添加空格或随机内容（来自语言模型）。采用肯德尔τ与排名偏倚重叠（RBO）等相似性度量，评估输出是否保持关系一致性。在MovieLens数据集上使用GPT3.5进行实验的结果显示，肯德尔τ与RBO的相似度显著下降，表明需为基于大模型的推荐系统引入更全面的评估框架。
原文摘要 · Abstract (English)
With the rise of Large Language Models (LLMs) such as ChatGPT, researchers have been working on how to utilize the LLMs for better recommendations. However, although LLMs exhibit black-box and probabilistic characteristics (meaning their internal working is not visible), the evaluation framework used for assessing these LLM-based recommender systems (RS) are the same as those used for traditional recommender systems. To address this gap, we introduce the metamorphic testing for the evaluation of GPT-based RS. This testing technique involves defining of metamorphic relations (MRs) between the inputs and checking if the relationship has been satisfied in the outputs. Specifically, we examined the MRs from both RS and LLMs perspectives, including rating multiplication/shifting in RS and adding spaces/randomness in the LLMs prompt via prompt perturbation. Similarity metrics (e.g. Kendall tau and Ranking Biased Overlap(RBO)) are deployed to measure whether the relationship has been satisfied in the outputs of MRs. The experiment results on MovieLens dataset with GPT3.5 show that lower similarity are obtained in terms of Kendall $τ$ and RBO, which concludes that there is a need of a comprehensive evaluation of the LLM-based RS in addition to the existing evaluation metrics used for traditional recommender systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。