arXiv:2509.21856cs.CL2025-09被引 3

首个评估多轮长文本问答的基准,专测大模型在真实对话中的知识准确性。

KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues

  • 构建动态对话生成机制,模拟真实多轮问答场景。
  • 发现对话越长,模型事实准确率下降、回答越啰嗦。
  • 验证检索增强生成能显著提升多轮问答的事实性。

多轮长文本问答(MT-LFQA)是大语言模型在知识密集型领域的重要应用。然而现有基准多局限于单轮对话,而多轮对话评测通常关注其他能力而非知识性事实。为此,我们提出首个系统评估多轮长文本问答的基准——KnowMT-Bench,覆盖医学、金融、法律等知识密集领域。该基准采用动态评估设置:模型根据逻辑递进的问题序列自动生成多轮对话历史,再以人工验证的自动化流程评估最终轮次回答的事实性与信息效率。实验发现,随着对话轮次增加,模型因自生对话上下文噪声导致事实能力下降,且回答趋于冗长。进一步研究显示,检索增强生成(RAG)可有效缓解甚至逆转事实性能退化。这些结果凸显了本基准在评估和提升大模型真实场景下对话知识能力的重要性。代码已开源。

原文摘要 · Abstract (English)

Multi-Turn Long-Form Question Answering (MT-LFQA) is a key application paradigm of Large Language Models (LLMs) in knowledge-intensive domains. However, existing benchmarks are limited to single-turn dialogue, while multi-turn dialogue benchmarks typically assess other orthogonal capabilities rather than knowledge-intensive factuality. To bridge this critical gap, we introduce \textbf{KnowMT-Bench}, the \textit{first-ever} benchmark designed to systematically evaluate MT-LFQA for LLMs across knowledge-intensive fields, including medicine, finance, and law. To faithfully assess the model's real-world performance, KnowMT-Bench employs a dynamic evaluation setting where models generate their own multi-turn dialogue histories given logically progressive question sequences. The factual capability and information delivery efficiency of the \textit{final-turn} answer are then evaluated using a human-validated automated pipeline. Our experiments reveal that multi-turn contexts degrade performance: factual capability declines due to the contextual noise from self-generated histories, while information efficiency drops as models become more verbose with increasing dialogue length. We then investigate mitigation strategies, demonstrating that retrieval-augmented generation (RAG) can effectively alleviate and even reverse this factual degradation. These findings underscore the importance of our benchmark in evaluating and enhancing the conversational factual capabilities of LLMs in real-world knowledge-intensive applications. Code is available at \href{https://github.com/hardenyu21/KnowMT-Bench}{\textcolor{cyan}{\texttt{KnowMT-Bench}}}.

多轮问答知识推理大模型评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。