arXiv:2605.09850cs.CVcs.AI2026-05

发现注意力残差模型的路由信息无法稳定提升校准效果

Probing Routing-Conditional Calibration in Attention-Residual Transformers

论文配图:Probing Routing-Conditional Calibration in Attention-Residual Transformers
图 1 · 摘自论文原文
  • 通过匹配置信度的诊断框架,检验路由痕迹对校准的影响
  • 30组置换检验仅1组在α=0.05下拒绝零假设,结果不稳健
  • 路由相关校准增益实为混淆因素所致,需严格对照验证

后验校准通常仅基于对数几率或softmax置信度评估,尽管路由增强架构在预测中引入样本特定的内部路由轨迹,并声称其与校准相关的不确定性有关。我们提出一个基本问题:这些轨迹是否能在置信度之外提供稳定的路由条件性校准证据?我们在注意力残差变压器(AR, Kimi Team, 2026)中通过匹配置信度的诊断套件进行研究,该套件按路由导出状态对样本分层,比较子组差距与组内路由置换零模型,并评估仅辅助特征不同的匹配后验探测器。在完成的全部AR运行中,标量路由汇总未能提供稳定的路由条件性校准偏差证据:加权差距保持较小或依赖种子;30组组内置换检验中仅有1组在α=0.05下拒绝条件零假设(仅在一个种子下成立,跨种子不稳定)。一种基于置信度和路由深度方差的最小二维Nadaraya-Watson探测器(AR-CondCal),其表现位于匹配置信度仅模型和预测熵控制的种子变异带内,且未可靠改善最差路由三分位的ECE。对完整向量的MLP(含$ c, H_1, \\-, H_L $)看似优于线性置信度基线,但一旦引入容量匹配的置信度仅模型作为对照,其性能优势即消失;打乱路由路径也能达到类似表现。在此AR设置中看似存在的路由感知校准增益,不应被解读为内部状态校准,除非匹配置信度、带宽、容量和置换控制排除了常见混淆因素。

原文摘要 · Abstract (English)

Post-hoc calibration is usually evaluated as a function of logits or softmax confidence alone, even as routing-augmented architectures increasingly accompany predictions with sample-specific internal routing traces and pair them with claims of calibration-relevant uncertainty. We ask a basic question: do these traces provide stable routing-specific evidence for post-hoc calibration beyond confidence? We study this in Attention-Residual transformers (Kimi Team, 2026) through a matched-confidence diagnostic suite that stratifies examples by routing-derived state, compares subgroup gaps against within-bin routing-permutation nulls, and evaluates matched post-hoc probes differing only in their auxiliary feature. Across our completed AR runs, scalar routing summaries do not provide stable evidence of routing-conditional miscalibration: weighted gaps remain small or seed-sensitive, and only $1$ of $30$ within-bin permutation tests rejects the conditional-null at $α=0.05$ (only on one seed; not stable across seeds in that cell). AR-CondCal, a minimal $2$-D Nadaraya--Watson probe on confidence and routing-depth variance, lies within the seed-variance band of matched confidence-only and predictive-entropy controls and does not reliably improve worst-routing-tertile ECE; bandwidth-sensitivity checks (Scott multiples, CV-NLL, global-ECE oracle) do not change this. A full-vector MLP over $(c, H_1, \ldots, H_L)$ can appear to improve over a linear confidence baseline, but the apparent gain disappears once a capacity-matched confidence-only MLP is included as a control, and shuffled routing profiles achieve comparable performance. Apparent routing-aware calibration gains in this AR setting should not be read as internal-state calibration until matched-confidence, bandwidth, capacity, and permutation controls rule out common confounds.

模型校准注意力机制路由追踪可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。