arXiv:2608.26846cs.CLcs.AI2026-08

校准信心能提升回答准确率,但可能降低覆盖率,需额外评估检索价值。

Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay

  • 用匹配轨迹重放对比原始信心与校准后信心的决策效果。
  • 校准使承诺回答的准确率最高提升41个百分点,但检索使用增加。
  • 校准无法预测单次检索是否有益,需独立评估信息价值。

交互式语言模型代理利用信心信号决定立即回答、检索额外证据或延迟决策。然而,信心通常孤立评估,未考察其触发行为的轨迹级后果。本文提出匹配轨迹重放协议,固定候选答案状态、证据点、预算和行动成本,用于比较信心到动作的映射。在Mistral、GPT和Qwen模型上,基于HotpotQA和MuSiQue数据集的多跳问答系统中,数值承诺阈值相同时,校准改变了代理最终承诺回答的问题。在所有六组模型-数据集组合中,校准使承诺答案的准确率最高提升41个百分点。但可能降低覆盖率并增加检索使用。整体准确率在HotpotQA上最高提升15个百分点,但在MuSiQue上最多下降17个百分点。这些变化反映更谨慎、低风险的运行点,而非答案质量或信心排序改善。事前拟合的校准映射在检索深度1和2时提升保留校准度,但在深度3时对所有三模型均劣于原始信心。平均而言,额外证据有帮助,但聚合效应不能说明信心能否识别哪些个体案例会从另一次检索中获益。结果表明,校准可使承诺风险可解释,但无法估计另一次检索的预期收益。因此,检索仍需独立的价值信息或效用估计。评估应报告保留校准度、风险-覆盖率和检索成本。

原文摘要 · Abstract (English)

Interactive language-model agents use confidence signals to decide whether to answer immediately, retrieve additional evidence (from memory or external knowledge), or defer. Yet confidence is usually evaluated in isolation, without measuring the trajectory-level consequences of the actions it triggers. We propose matched trajectory replay, a controlled protocol for comparing confidence-to-action mappings. The protocol holds candidate answer states, evidence points, budgets, and action costs fixed. We use it to compare raw verbalized confidence with post-hoc isotonic calibration in a multi-hop question-answering system using Mistral, GPT, and Qwen models on HotpotQA and MuSiQue datasets. At the same numerical commitment threshold, calibration changes which questions agents ultimately commit to answering. Across all six model-dataset pairs, it increases accuracy among committed answers by up to 41 percentage points. However, it can reduce coverage and increase retrieval use. Overall accuracy improves by up to 15 percentage points on HotpotQA but falls by up to 17 percentage points on MuSiQue. These effects reflect a shift to a more selective, lower-risk operating point, not improved answers or confidence ranking. A calibration map fitted before retrieval improves held-out calibration through retrieval depths one and two, but is worse than raw confidence at depth three for all three models. Additional evidence helps on average, but this aggregate effect does not establish whether confidence identifies which individual episodes will benefit from another retrieval. Taken together, these results show that calibration can make commitment risk interpretable, but it does not estimate the expected benefit of another retrieval. Retrieval therefore requires a separate value-of-information or utility estimate. Evaluations should report held-out calibration, risk-coverage, and retrieval cost.

信心校准检索策略多跳问答

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。