让大模型自己判断何时该查资料,提升问答效率与准确率。
Beyond Self-Knowledge: Propagating Uncertainty Across Reasoning and Retrieval in LLMs

- 用模型自评置信度决定是否调用检索,低信心时才查资料。
- 在6个数据集上平均F1达0.483,比始终检索高0.016,且减少20.4%检索量。
- 适合追求高效精准问答的场景,尤其关注推理成本与性能平衡者。
检索增强生成提升了知识密集型问答表现,但盲目检索会引入无关证据并增加计算开销。本文研究黑盒大模型输出的置信度是否可作为检索路由的有效信号。提出BeyondUncertainty方法:先获取结构化初步答案与置信度估计,再根据在保留验证集上选定的模型特异性阈值(测试前冻结)判断是否检索。低置信度问题触发前5个TF-IDF检索并二次生成答案,高置信度问题则直接返回初步答案。在六个QA基准、三个模型族和三种检索策略下评估27,000个策略实例。BeyondUncertainty实现0.483的平均分词级F1,优于始终检索的0.467和不检索的0.401,同时相比始终检索减少20.4%检索段落数。在每数据集-模型单元内匹配检索问题数量时,其在18组中的17组优于事后随机分配,平均提升0.024 F1。尽管置信度绝对值校准不佳,但能中等预测问题级检索收益(AUROC=0.628)。然而额外探测使总令牌消耗增加28.2%,揭示了更精准检索与端到端效率之间的权衡。
原文摘要 · Abstract (English)
Retrieval-augmented generation improves knowledge-intensive question answering, but indiscriminate retrieval can introduce irrelevant evidence and unnecessary computation. We investigate whether verbalized confidence from black-box language models can serve as an actionable signal for retrieval routing. Our method, BeyondUncertainty, first elicits a structured provisional answer and confidence estimate, then applies a model-specific threshold selected on held-out validation data and frozen before test evaluation. Low-confidence questions receive top-5 TF-IDF retrieval followed by a second answer call, whereas high-confidence questions return the provisional answer directly. We evaluate 27,000 policy instances across six QA benchmarks, three model families, and three retrieval policies. BeyondUncertainty achieves 0.483 mean token-level F1, compared with 0.467 for always retrieval and 0.401 for no retrieval, while reducing retrieved passages by 20.4\% relative to always retrieval. When matched on the number of questions routed to retrieval within each dataset-model cell, it outperforms a post-hoc random allocation in 17 of 18 settings, with an average gain of 0.024 F1. Although poorly calibrated as an absolute probability, probe uncertainty modestly predicts question-level retrieval benefit (AUROC = 0.628). However, the additional probe increases total token usage by 28.2\%, revealing a trade-off between more selective evidence acquisition and end-to-end token efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。