用可解释的智能体操作替代黑箱检索,提升长文档问答准确率。
Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations

- 设计可回溯的三步智能体操作:词法搜索、结构导航、限定范围读取
- 在政府财务报告上准确率58.8%,远超传统检索的15.7%(p=2e-5)
- 适合需要可解释性与高精度的金融/审计类文档分析场景
长文档的检索增强生成普遍采用分块、嵌入、取前k近邻的模式。我们指出,对于财务报表、审计报告等结构化强的文档,该设计存在根本缺陷:某780页政府财务报告中86.8%内容为表格行,数千个近乎相同的数值共存于同一嵌入空间,且数值单位常由上方第13行的标题决定——分块边界常将数字与其单位分离,导致量级错误达两个数量级。虽表结构感知分块器可解决单位问题,但仍有27%-30%数值块缺失年度标题。本文提出READ(无嵌入可靠智能体文档搜索),通过三个确定性操作(归一化词法搜索、结构导航、受限跨度读取)在模型上下文协议下运行,使推理轨迹可复现。在51个验证问题上,READ准确率达58.8%,远超密集检索的15.7%(p_Holm = 2×10^-5),调优后达35.3%,仍领先23.5个百分点(p_Holm = 0.017)。使用相同循环但仅换检索工具的代理准确率仅27.5%,表明性能提升源于接口设计而非迭代本身。同时发现,BM25与READ无统计差异,说明结果差异来自嵌入式与无嵌入检索,而非智能体与词法搜索。
原文摘要 · Abstract (English)
Retrieval-augmented generation over long documents is dominated by one design: chunk the text, embed the chunks, and surface the top-k nearest neighbours of the query. We argue that for an important class of documents -- financial statements, audit reports, regulatory returns -- this design is structurally unsound, and we make the argument measurable. On a 780-page government financial report, 86.8% of content lines are table rows, thousands of near-identical figures compete in one embedding space, and a figure inherits its unit from a header a median of 13 lines above it -- so a chunk boundary routinely separates a number from whether it is in lakh or crore, an error of two orders of magnitude. A table-aware chunker built as a steelman fixes the unit problem but leaves 27-30% of numeric chunks with no fiscal-year header at every chunk size we tried. We propose READ (Reliable Embedding-free Agentic Document-search), in which an agent reads the raw document through three deterministic operations -- normalized lexical search, structural navigation, and bounded span reads -- exposed over the Model Context Protocol, so a trajectory is a replayable audit trail, not an opaque similarity score. On 51 verified questions READ answers 58.8% against dense retrieval's 15.7% (p_Holm = 2 x 10^-5) -- or 35.3% tuned, which READ still leads by 23.5 points (p_Holm = 0.017). An agent given the same loop but a top-k tool reaches only 27.5%, locating the gain in the interface rather than in iteration. We also report what the evidence does not support: BM25 is statistically indistinguishable from READ, so our result separates embedding-based from embedding-free retrieval, not agentic from lexical search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。