构建含歧义与多跳推理的评测集，检验大模型答全能力。
DEEPAMBIGQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness
- 自动生成融合知识图谱的歧义多跳问题
- 3600题中半数需解决名称歧义，顶尖模型准确率仅13%
- 适合评估大模型信息搜集与答案完整性
大型语言模型（LLMs）结合搜索工具在开放域问答中表现优异，但常无法完整回答复杂问题，如“哪位《热》电影的演员获得过奥斯卡奖？”该问题需区分同名影片并跨演员群体推理。现有评测基准极少同时考察这两点。为此，我们提出DeepAmbigQAGen自动数据生成管道，基于文本语料库和链接知识图谱生成自然、可验证的问题，系统性引入名称歧义与多步推理。基于此构建了DeepAmbigQA数据集，包含3600个需多跳推理的问题，其中一半明确涉及名称歧义。实验表明，即使是最先进的GPT-5模型，在歧义问题上精确匹配率也仅达0.13，在非歧义问题上为0.21。结果凸显了对更鲁棒问答系统以提升信息获取与答案完整性的迫切需求。
原文摘要 · Abstract (English)
Large language models (LLMs) with integrated search tools show strong promise in open-domain question answering (QA), yet they often struggle to produce complete answer set to complex questions such as Which actor from the film Heat won at least one Academy Award?, which requires (1) distinguishing between multiple films sharing the same title and (2) reasoning across a large set of actors to gather and integrate evidence. Existing QA benchmarks rarely evaluate both challenges jointly. To address this, we introduce DeepAmbigQAGen, an automatic data generation pipeline that constructs QA tasks grounded in text corpora and linked knowledge graph, generating natural and verifiable questions that systematically embed name ambiguity and multi-step reasoning. Based on this, we build DeepAmbigQA, a dataset of 3,600 questions requiring multi-hop reasoning and half of them explicit name ambiguity resolving. Experiments reveal that, even state-of-the-art GPT-5 show incomplete answers, achieving only 0.13 exact match on ambiguous questions and 0.21 on non-ambiguous questions. These findings highlight the need for more robust QA systems aimed at information gathering and answer completeness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。