通过分析证据构建失败原因，提升大模型问答的可靠性
AuthTrace: Diagnosing Evidence Construction in Thematically Dense Single-Author Corpora
- 基于主题密集的单作者语料设计诊断基准，模拟真实写作风格
- 发现证据召回率是答案正确性的强预测指标（r=0.96），多数错误源于漏检证据
- 提出‘依赖深度’分析框架，可定位不同检索范式失效的具体环节
证据构建——决定生成前哪些段落进入语言模型的阶段——目前按范式逐一评估，使从业者无法系统诊断组织策略在何处、为何失败。我们提出 AuthTrace，一个基于主题密集单作者语料库的诊断基准，其中近似干扰项与目标证据共享风格、主题和词汇。AuthTrace 提供明确引用证据、精确的源文档归因标注，以及统一的打包级协议，衡量证据召回率、证据准确率与答案正确性。以‘依赖深度’（fan-in）——支持答案所需的源文档数量——作为主要诊断轴，实现对检索、记忆、图结构及结构化证据范式的可控对比。在两个 QA 模型上评估八个系统，发现证据召回率是主阅读-评判对下答案正确性的最强预测因子（r = 0.96）；多数失败源于证据缺失而非答案合成。依赖深度进一步揭示范式特异性退化模式：扁平检索的性能衰减速度比主题组织证据构建快2-3倍。结果表明，依赖深度分解是一种可复用的诊断工具，能识别证据构建系统失效位置，并匹配特定工作负载的最佳范式。
原文摘要 · Abstract (English)
Evidence construction--the stage that determines which passages reach the language model before generation begins--is evaluated paradigm by paradigm, leaving practitioners with no principled way to diagnose which organization strategy fails, where, or why. We introduce AuthTrace, a diagnostic benchmark built on thematically dense single-author corpora where near-miss distractors share style, topic, and vocabulary with the required evidence. AuthTrace provides explicit quoted evidence, exact fan-in annotation, and a unified pack-level protocol measuring evidence recall, evidence precision, and answer correctness. A fan-in gradient--the number of source documents required to support the answer--serves as the primary diagnostic axis, enabling controlled comparison across retrieval, memory, graph, and structured-evidence paradigms. Evaluating eight systems across two QA models, we find that evidence recall is the strongest observed predictor of answer correctness under the primary reader-judge pair (r = 0.96); most failures stem from missing evidence rather than answer synthesis. Fan-in further exposes paradigm-specific collapse patterns: flat retrieval degrades 2-3x faster than thematically organized evidence construction. These results show fan-in decomposition to be a reusable diagnostic lens for identifying where evidence-construction systems fail and which paradigm best serves a given workload.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。