用真人对决评测对话推荐系统，结果更可靠。
CRS Arena: Crowdsourced Benchmarking of Conversational Recommender Systems
- 用户两两对比匿名推荐系统，实时投票选优劣。
- 在开闭源平台测试，排名相关性高达0.92以上。
- 开源474条对话数据与评分，适合评测研究者使用。
我们提出CRS Arena，一个基于人类反馈的可扩展对话推荐系统（CRS）评测平台。该平台以成对对抗的形式展示匿名的对话推荐系统，用户依次与两个系统交互后选择胜者或平局。平台收集对话记录与用户反馈，为CRS的可靠评估与排序提供基础。我们在开放与封闭的众包平台上开展实验，结果表明两种设置下系统排名高度相关（相关系数>0.92），且对话特征相似。我们发布CRSArena-Dial数据集，包含474条对话及其用户反馈，并基于Elo评分系统给出初步系统排名。平台网址：https://iai-group-crsarena.hf.space/。
原文摘要 · Abstract (English)
We introduce CRS Arena, a research platform for scalable benchmarking of Conversational Recommender Systems (CRS) based on human feedback. The platform displays pairwise battles between anonymous conversational recommender systems, where users interact with the systems one after the other before declaring either a winner or a draw. CRS Arena collects conversations and user feedback, providing a foundation for reliable evaluation and ranking of CRSs. We conduct experiments with CRS Arena on both open and closed crowdsourcing platforms, confirming that both setups produce highly correlated rankings of CRSs and conversations with similar characteristics. We release CRSArena-Dial, a dataset of 474 conversations and their corresponding user feedback, along with a preliminary ranking of the systems based on the Elo rating system. The platform is accessible at https://iai-group-crsarena.hf.space/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。