通过轨迹几何分析，更高效精准地评估大模型推理的置信度。
Measuring Black-Box Confidence via Reasoning Trajectories: Geometry, Coverage, and Verbalization
- 将推理过程建模为滑动窗口轨迹，用单参数软最大值衡量收敛性。
- 在6个基准上融合几何、覆盖和语言表达信号，性能优于传统方法（提升0.075 AUC）。
- 无需模型内部信息，适合对黑盒模型进行可信度评估的研究者使用。
可靠的置信度估计对于通过纯文本API安全部署链式思维（CoT）推理至关重要。现有主流黑盒基线——基于K个样本的自一致性方法——计算开销线性增长且忽略推理轨迹的几何结构。本文提出一种黑盒轨迹置信度评分：将CoT嵌入为滑动窗口轨迹，用单参数软最大值度量其向外部答案锚点的收敛程度，无需对数几率、隐藏状态或监督校准器。在MedQA-USMLE、GPQA Diamond和MMLU-Pro六个（基准，推理器）组合上，使用Gemini 3.1 Pro与Claude Sonnet 4.6，在K=4时融合该得分与覆盖及语言化置信通道，相较K=8的自一致性在全部6个设置中实现帕累托改进（中位AUC 0.78 vs 0.71，ΔAUC=+0.075）。固定选择控制组（+0.060）与E5跨嵌入器复现规则排除了答案切换与单一供应商偏差。几何信号在多数基准中于倒数第二窗口达到峰值，而在GPQA Diamond的终止窗口发生反转。三种无引导范式将黑盒置信度分解为判别者驱动的覆盖先验（C）、轨迹内几何（G）与条件语言表达通道（V）。在18个基准×推理器×提议者设置中，C与G在全部18个和16个场景中提供独立信号，而V在6个场景中贡献残差信号。将判别者从GPT-5-mini更换为Claude Sonnet 4.6后，仅影响极小（|Δ|≤0.013），且覆盖仅偏移±0.02（κ=0.82）。融合策略在17/18场景中优于单一通道（中位AUC 0.78，最高0.92）。
原文摘要 · Abstract (English)
Reliable confidence estimation enables safe deployment of chain-of-thought (CoT) reasoning through text-only APIs. Yet the dominant black-box baseline, self-consistency over K samples, is linearly expensive and ignores the geometry of the trace. We propose a black-box trajectory-confidence score: we embed a CoT as a sliding-window trajectory and measure its convergence to external answer anchors with a one-parameter softmax. The method needs no logits, hidden states, or supervised calibrators. Across six (benchmark, reasoner) settings on MedQA-USMLE, GPQA Diamond, and MMLU-Pro with Gemini 3.1 Pro and Claude Sonnet 4.6, fusing this score with coverage and verbalized-confidence channels at K=4 yields Pareto improvements over self-consistency at K=8 in 6/6 settings (median AUC 0.78 vs 0.71, deltaAUC=+0.075). A fixed-pick control (+0.060) and E5 cross-embedder replication rule out answer switching and single-vendor artifacts. Geometry peaks in the penultimate window across benchmarks and reasoners, and inverts at the terminal window on GPQA Diamond. Three unscaffolded regimes separate black-box confidence into a judge-mediated Coverage prior (C), within-trace Geometry (G), and a conditional Verbalization channel (V). Across 18 benchmark x reasoner x proposer settings, C and G provide independent signal in 18/18 and 16/18, while V contributes residual signal in 6/18. Swapping the judge from GPT-5-mini to Claude Sonnet 4.6 leaves G-only AUC unchanged (|delta|<=0.013) and shifts C-only AUC by at most +/-0.02 (kappa=0.82). Fusion beats the best single channel in 17/18 settings (median AUC 0.78, max 0.92).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。