构建新问答任务，让模型从线索中推理未明说的答案。
Inferential Question Answering
- 设计新任务Inferential QA，要求从隐含线索推断答案
- 在7401个问题上测试，现有模型性能普遍偏低
- 揭示当前系统难处理间接证据推理，适合研究推理能力者
尽管已有大量关于问答系统的研究，但多数工作聚焦于可直接提取或生成答案的题目。然而，部分问题需要推理——即从已有信息中推导出未明确陈述的答案。为此，本文提出Inferential QA新任务，挑战模型从仅提供线索的支撑段落中推断答案。我们构建QUIT数据集，包含7,401个问题和240万条段落，基于高收敛性的人工与机器生成提示，并通过基于大模型的答案可回答性评估与人工验证，在三个相关性层级上进行标注。对检索器、重排序器及大模型阅读器的全面评估表明：传统问答有效的方法在推理型问答中表现不佳——检索器效果差，重排序器增益有限，微调带来不一致提升。甚至面向推理的大模型也未能超越小型通用模型。这些发现表明，当前问答流程尚未具备处理基于推理的能力。Inferential QA确立了一类新的问答任务，推动系统向基于间接文本证据的理解与推理迈进。
原文摘要 · Abstract (English)
Despite extensive research on a wide range of question answering (QA) systems, most existing work focuses on answer containment-i.e., assuming that answers can be directly extracted and/or generated from documents in the corpus. However, some questions require inference, i.e., deriving answers that are not explicitly stated but can be inferred from the available information. We introduce Inferential QA -- a new task that challenges models to infer answers from answer-supporting passages which provide only clues. To study this problem, we construct QUIT (QUestions requiring Inference from Texts) dataset, comprising 7,401 questions and 2.4M passages built from high-convergence human- and machine-authored hints, labeled across three relevance levels using LLM-based answerability and human verification. Through comprehensive evaluation of retrievers, rerankers, and LLM-based readers, we show that methods effective on traditional QA tasks struggle in inferential QA: retrievers underperform, rerankers offer limited gains, and fine-tuning provides inconsistent improvements. Even reasoning-oriented LLMs fail to outperform smaller general-purpose models. These findings reveal that current QA pipelines are not yet ready for inference-based reasoning. Inferential QA thus establishes a new class of QA tasks that move towards understanding and reasoning from indirect textual evidence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。