发现表格检索中语义相似却答案不可用的漏洞,提出诊断基准并改进方法。
Semantically Similar, Logically Distinct: Diagnosing the Semantic-Answerability Gap in Table RAG

- 构建同构表格对,诊断检索模型在答案可用性上的盲区。
- 检索性能从0.755降至0.330,暴露语义与答案可用性的脱节。
- 轻量级重排序方法提升准确率至57.4%,证明验证环节缺失是主因。
表格是检索增强生成(RAG)中的关键知识源,但检索到的表格可能缺乏回答问题所需证据,这一现象称为答案可用性。尽管答案可用性关乎源是否包含足够证据,但以语义相关性优化的检索模型在单源情况下也无法保证其可用性,造成根本性错配。为此,我们提出TCR-Bench,一个针对表内容级答案可用性的诊断基准,基于具有高度相似模式但细微内容差异的同源表格。在TCR-Bench上,所评估的密集检索器持续表现出语义-答案可用性差距:常能检索到正确表格组,却难以定位其中唯一可回答问题的表格,导致问答性能从0.755(理想情况)下降至0.330(前5个检索结果)。分析表明,该差距与语义累积、模式级线索依赖及弱行-列绑定有关。作为诊断探针,我们测试了一种轻量级两阶段管道——答案可用性感知重排序(AAR),通过直接判断查询与表格的答案可用性来修复缺陷:其将最高优先目标表检索准确率从18.2%提升至57.4%,显著增益说明多数失败源于缺乏答案可用性验证步骤,而非模型容量本身的局限。
原文摘要 · Abstract (English)
Tables are a critical knowledge source in retrieval-augmented generation (RAG), but a retrieved table may lack sufficient evidence to answer a query, a property we call answerability. While answerability broadly concerns whether a source or collection of sources contains sufficient evidence, retrieval models optimized for semantic relevance do not guarantee it even in the single-source case, creating a fundamental mismatch. To study this, we introduce TCR-Bench, a diagnostic benchmark for Table Content-level Answerability in RAG, built around sibling tables, i.e., tables with highly similar schemas but subtle content differences. On TCR-Bench, the dense retrievers we evaluate persistently exhibit a Semantic-Answerability Gap: they often retrieve the correct sibling group yet struggle to pinpoint the uniquely answerable table within it, dropping QA performance from 0.755 (oracle) to 0.330 (top-5 retrieved). Our analysis suggests this gap is associated with semantic accumulation, schema-level cue dependence, and weak row-column binding. As a diagnostic probe into the source of this gap, we test whether a lightweight two-stage pipeline, Answerability-Aware Reranking (AAR), applying direct query-table answerability judgment, can recover performance: it raises top-1 target retrieval from 18.2% to 57.4%, and this large gain is itself evidence that much of the observed failure reflects a missing answerability verification step, rather than an inherent limitation of model capacity alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。