提出可定位多跳问答中证据缺失步骤的混合检查器StepGap。
StepGap: A Hybrid NLI-LLM Checker for Step-Level Evidence-Gap Detectionin Multi-Hop Question Answering
- 结合NLI与大模型构建分步决策树，识别三类证据问题。
- 在82个问题上达到72.0的sF1，优于纯大模型的错误掩盖效果。
- 适合用于改进大模型问答的可解释性与修复策略设计。
我们提出StepGap，一种融合NLI与大模型的混合决策树，用于检测多跳问答中的步骤级证据缺失，并输出三类标签：矛盾声明（CC）、无关证据（IE）和缺失桥接（MB），每类对应具体修复动作。在82个多跳问题（181个标注步骤，κ=0.704）上，StepGap取得sF1=72.0，处于大模型基线（70.1）的置信区间内，但结构更可分解：移除任一阶段均导致F1下降，而四个大模型阶段中有三个移除后反而提升F1——表明存在竞争性误差抵消现象。我们进一步揭示‘Q-F1陷阱’：仅关注问题级F1会因过度标记步骤而被机械抬高，因此步骤级F1才是必要诊断指标。作为类型化GRPO奖励，StepGap将Qwen2.5-7B-Instruct的Exact Match从32.1±0.3提升至35.4±0.9（三种子），单次运行相比匹配的Search-R1 GRPO复现提升+5.6 Avg EM。
原文摘要 · Abstract (English)
We present \textbf{StepGap}, a hybrid NLI-LLM decision tree that detects step-level evidence gaps in multi-hop QA and emits one of three typed labels: \textsc{Contradicted Claim} (CC), \textsc{Irrelevant Evidence} (IE), or \textsc{Missing Bridge} (MB), each tied to a concrete repair action. On 82 multi-hop questions (181 annotated steps, $κ{=}0.704$), StepGap reaches sF1$=$72.0, within the bootstrap confidence interval of an LLM-only baseline (70.1) but with a more decomposable structure: every StepGap stage \emph{hurts} F1 when removed, while three of four LLM-only removals \emph{improve} F1 -- a sign of \emph{competing-error cancellation}, where internal stages mask each other's errors. We further expose a \emph{Q-F1 trap}: question-level F1 is mechanically inflated by checkers that flag every step, making step-level F1 the necessary diagnostic. Used as a typed GRPO process reward, StepGap improves Qwen2.5-7B-Instruct Exact Match from $32.1{\pm}0.3$ to $35.4{\pm}0.9$ across three seeds, with the single-run comparison showing a $+5.6$ Avg EM gain over the matched Search-R1 GRPO reproduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。