SimLab是首个云端对话信息获取评估平台，让研究者可重复测试对话系统和用户模拟器。
SimLab: A Platform for Simulation-based Evaluation of Conversational Information Access Systems
- 基于云平台实现对话系统与用户模拟器的标准化评估
- 在电影推荐场景中完成首次仿真评估任务
- 支持可复现实验，适合对话系统与用户模拟研究者使用
对话式信息获取（CIA）系统的发展受限于难以开展可复现的评估实验。虽然用户模拟提供了一条有前景的解决方案，但缺乏支撑该范式的基础设施和工具仍是主要障碍。为此，我们推出了SimLab——首个基于云的平台，为社区提供集中化的基准测试环境，用于在可控、可复现条件下评估对话系统与用户模拟器。我们明确了此类平台的需求，并提出了通用基础设施方案。随后，我们展示了SimLab的初步设计与实现，并通过一次电影推荐场景中的仿真评估任务展示其功能。此外，我们讨论了平台的可持续性及未来发展机会，诚邀社区共同推进CIA与用户模拟领域的发展。
原文摘要 · Abstract (English)
Progress in conversational information access (CIA) systems has been hindered by the difficulty of evaluating such systems with reproducible experiments. While user simulation offers a promising solution, the lack of infrastructure and tooling to support this evaluation paradigm remains a significant barrier. To address this gap, we introduce SimLab, the first cloud-based platform providing a centralized solution for the community to benchmark both conversational systems and user simulators in a controlled and reproducible setting. We articulate the requirements for such a platform and propose a general infrastructure to meet them. We then present the design and implementation of an initial version of SimLab and showcase its features through an initial simulation-based evaluation task in conversational movie recommendation. Furthermore, we discuss the platform's sustainability and future opportunities for development, inviting the community to drive further progress in the fields of CIA and user simulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。