评测大模型在新知识和冷门知识上的多跳问答能力
MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge
- 构建了涵盖新知识与长尾知识的多跳问答测试集
- 22个顶尖大模型在新/冷门知识上表现普遍不佳
- 适合研究大模型推理与知识泛化能力的学者使用
大语言模型在多种推理任务中表现出色，但在涉及新知识或长尾知识的复杂多跳查询上仍面临显著挑战。现有基准未能充分覆盖此类问题。为此，我们提出MINTQA（针对新知识与长尾知识的多跳问答基准），全面评估大模型在四个关键维度的能力：问题处理策略、子问题生成、检索增强生成，以及迭代式或动态分解与检索。MINTQA包含10,479个用于评估新知识的问答对和17,887个用于评估长尾知识的问答对，每道题均配有对应的子问题与答案。我们对22个前沿大模型进行系统评估，发现它们在处理复杂知识库查询时存在明显局限，尤其在新知识和小众知识方面表现较差。研究结果揭示了关键挑战，并为提升多跳推理能力提供了洞见。MINTQA基准已开源：https://github.com/probe2/multi-hop/
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks but face significant challenges with complex, knowledge-intensive multi-hop queries, particularly those involving new or long-tail knowledge. Existing benchmarks often fail to fully address these challenges. To bridge this gap, we introduce MINTQA (Multi-hop Question Answering on New and Tail Knowledge), a comprehensive benchmark to evaluate LLMs' capabilities in multi-hop reasoning across four critical dimensions: question handling strategy, sub-question generation, retrieval-augmented generation, and iterative or dynamic decomposition and retrieval. MINTQA comprises 10,479 question-answer pairs for evaluating new knowledge and 17,887 pairs for assessing long-tail knowledge, with each question equipped with corresponding sub-questions and answers. Our systematic evaluation of 22 state-of-the-art LLMs on MINTQA reveals significant limitations in their ability to handle complex knowledge base queries, particularly in handling new or unpopular knowledge. Our findings highlight critical challenges and offer insights for advancing multi-hop reasoning capabilities. The MINTQA benchmark is available at https://github.com/probe2/multi-hop/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。