构建首个多轮检索生成评测基准，揭示大模型在真实对话中的短板。
MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems
- 人工构建110个多轮对话，覆盖4个领域，每轮平均7.7次交互。
- 顶尖RAG系统在后期对话和非独立问题上表现仍差，准确率不足预期。
- 适合评估多轮对话中检索与生成协同能力的研究者使用。
检索增强生成（RAG）已成为大语言模型的重要任务。然而，在包含先前对话上下文的多轮RAG对话中评估系统性能，是一项重要但常被忽视的任务，面临诸多额外挑战。我们提出MTRAG：一个端到端的人工生成多轮RAG基准，涵盖多样现实属性，用于评估完整的RAG流程。MTRAG包含110个对话，平均每轮7.7次交互，覆盖四个领域，总计842项任务。我们还探索了通过合成数据和大模型作为裁判的自动化评估路径。人工与自动评估表明，即使最先进的LLM-RAG系统在MTRAG上也表现不佳。实验揭示了对强检索与生成系统的需求，尤其在处理后期对话、无法回答的问题、非独立问题及跨领域场景下。MTRAG已开源，地址为https://github.com/ibm/mt-rag-benchmark。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at https://github.com/ibm/mt-rag-benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。