arXiv:2607.24010cs.LG2026-07中稿 · KDD被引 1

提出预算感知评估框架,解决主动检索系统误判效率的问题。

When Should Active RAG Retrieve? A Budget-Aware Evaluation of Utility, Calibration, and Cost

论文配图:When Should Active RAG Retrieve? A Budget-Aware Evaluation of Utility, Calibration, and Cost
图 1 · 摘自论文原文
  • 将主动检索视为边际正确率提升的效用估计,分离评估维度
  • 发现检索错误不可忽略,且阈值迁移误差显著影响实际使用率
  • 建议报告效用前沿、成本分解等多维指标,而非单一准确率

主动RAG系统在生成过程中决定何时检索外部知识,属于预算敏感型代理RAG与自适应检索。然而现有评估常未明确定义运行点:两个系统均宣称使用50%证据预算,但实际使用率可能不同,更高准确率可能源于更宽松的预算,而非更优的检索策略。本文通过将主动检索重构为效用估计,仅当检索带来边际正确性提升时才具价值,从而区分三个被单一指标混淆的问题:触发得分是否能正确排序检索决策、历史校准阈值能否满足未来预算、触发端计算如何影响部署成本。我们提出精确的top-k效用前沿、可部署阈值前沿、保守预算前沿、伤害审计及成本分解等方法。在多跳问答数据集和开放指令模型上,检索伤害不可忽视,路由器排名随数据集和预算变化,名义阈值常偏离目标使用率,而简单的不确定性或检索分数基线常媲美学习得到的效用路由器。因此,预算感知的主动RAG评估应报告效用前沿、实际使用率、阈值迁移误差、伤害率与成本分解,而不仅是准确率。

原文摘要 · Abstract (English)

Active RAG systems decide when to retrieve external knowledge during generation, making them a budget-sensitive case of agentic RAG and self-adaptive retrieval. Yet evaluations often leave the operating point underspecified: two systems may both claim a 50% evidence-usage budget while realizing different held-out usage rates, so higher accuracy can reflect a looser budget rather than a better retrieval policy. We study budget-aware evaluation for Active RAG by recasting active retrieval as utility estimation, where retrieval is valuable only through its marginal correctness change over a no-retrieval answer. This view separates three questions that single-point evaluations conflate: whether trigger scores rank useful retrieval decisions, whether thresholds calibrated on past data meet future budgets, and how trigger-side computation changes deployment cost. We operationalize these questions with exact top-k utility frontiers, deployable threshold frontiers, conservative budget frontiers, harm audits, and cost decompositions. Across knowledge-intensive multi-hop QA datasets and open instruction models, retrieval harm is non-negligible, router rankings change across datasets and budgets, nominal thresholds can miss target usage, and simple uncertainty or retrieval-score baselines often rival learned utility routers. Budget-aware Active RAG evaluations should therefore report frontiers, realized usage, threshold-transfer error, harm rates, and cost decompositions alongside accuracy.

主动检索预算评估RAG效用估计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。