arXiv:2607.00725cs.CLcs.IR2026-07

提出新指标评估检索生成中答案是否保留,比传统召回更有效。

Recall Is Not Enough: A Reader-Context Diagnostic for Budget-Constrained Retrieval-Augmented Generation

论文配图:Recall Is Not Enough: A Reader-Context Diagnostic for Budget-Constrained Retrieval-Augmented Generation
图 1 · 摘自论文原文
  • 引入答案在上下文中的留存率作为新诊断指标
  • 该指标能解释4.6倍的准确率差异,超越召回率
  • 可指导高效打包算法设计,适合模型优化研究者

固定上下文预算下的检索增强生成面临选择难题:仅部分检索证据能放入读者上下文。当前标准指标recall@k基于检索结果评分,但读者实际使用的是打包后的上下文——一旦打包需丢弃信息,两者便脱节。本文提出“答案在上下文”(answer-in-context)诊断指标,衡量黄金答案是否保留在打包后上下文中,并主张预算型RAG应以该量为目标优化。该指标在三个多跳数据集上相比召回率额外提升Delta R² = 0.17–0.27;即使所有黄金证据均被检索到,打包是否保留答案仍使精确匹配率相差4.6倍。两项独立干预验证其中介作用:提升文档覆盖率但不改善答案留存率时准确率不变;提示压缩破坏答案片段会同时降低二者。进一步提出分级变体适用于自由格式答案。最后证明该诊断可行动:将读者上下文构建建模为带预算的子模最大化问题,所得打包器在三种阅读器、四种规模、四种预算下均优于现有方法(包括top-k截断与LLMLingua-2压缩),且在同等或更低令牌成本下表现更优。与手动调参的查询聚焦启发式相比,达到相当水平,仅在证据密度受限时胜出。整个过程由单一变量预测有效性。

原文摘要 · Abstract (English)

Retrieval-augmented generation under a fixed context budget forces a selection problem: only a fraction of the retrieved evidence fits in front of the reader. The field's standard metric, recall@k, is scored on the retrieved set, but the reader consumes the packed context - and once packing must discard evidence, the two come apart. We introduce answer-in-context, a diagnostic that measures whether a gold answer survives into the packed context, and argue it is the quantity budgeted RAG should be optimizing. It carries substantial information beyond retrieval, adding Delta R^2 = 0.17-0.27 over recall across three multi-hop datasets; even among questions where all gold was retrieved, whether packing keeps the answer separates exact match by 4.6x. Two independent interventions confirm the mediation: a packing change that raises document coverage without raising answer-in-context leaves accuracy flat, and prompt compression that destroys the answer span lowers both together. A graded variant extends the diagnostic to free-form answers, where no verbatim span exists. We then show the diagnostic is actionable. Casting reader-context construction as budgeted submodular maximization gives a packer that beats both deployed top-k truncation and LLMLingua-2 compression - across three reader families, four scales, and four budgets, at equal-or-lower token cost. Against a hand-tuned query-focused heuristic, which we show approximates the same objective, it reaches parity, winning outright only where evidence density is the binding constraint. Throughout, one variable predicts what helps and what cannot.

检索增强上下文打包模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。