arXiv:2604.24801cs.LGcs.AI2026-04

模型架构决定错误可监测性,影响监控系统能否发现自信却错误的预测。

Architecture Determines Observability of Transformers

  • 通过分析模型架构对激活信号的影响,揭示可观测性的根本来源。
  • 60.3%的激活探测信号被输出置信度覆盖,剩余信号依赖架构与训练过程。
  • 无需任务微调的通用探测器可捕获约1/8被置信度忽略的错误,适合模型可靠性研究。

自回归Transformer会做出自信却错误的预测,而输出置信度监控无法捕捉这些错误。激活监控仅在训练留下超越输出本身的信息信号时才有效,该信号是模型架构的固有属性。控制输出置信度后,14个模型中平均有60.3%的原始激活探测信号消失。原始信号主要由输出置信度构成,输出侧读取无法恢复残余信号。残余信号取决于架构和训练过程。在Pythia的受控训练中,相同宽度配置的模型早期均生成信号,但一个在收敛过程中保留信号,另一个则随困惑度持续下降而消除信号。能力与可观测性并非天然冲突。在独立训练的多个模型族中,该模式依然存在,即使信号崩溃点发生偏移。信号得以留存之处,监控系统可捕捉到置信度无法发现的错误。在下游问答任务中,一个基于WikiText训练、无任务微调的探测器,在20%的标记率下,可捕捉约1/8被置信度监控遗漏的自信错误。这些结果确立信号工程为训练阶段的重要设计维度,架构决定了可观测性的基础条件,训练则决定何种信号可被读取。

原文摘要 · Abstract (English)

Autoregressive transformers make confident errors that output-confidence monitoring cannot catch. Activation monitors catch them only when training leaves a decision-quality signal beyond what the output already exposes. This signal is an architectural property of the trained model, fixed upstream of any monitor. Controlling for output confidence removes 60.3% of the raw activation-probe signal on average across 14 models. Raw probe signal is mostly output confidence, and output-side readouts cannot recover the residual. What remains depends on architecture and training. In Pythia's controlled training, both matched-width configurations form the signal early. One preserves it through convergence while another erases it as perplexity continues to improve. Capability and observability are not inherently in tension. Across independently trained families this pattern persists, even as the collapse point shifts. Where the signal survives, monitoring catches what confidence cannot. On downstream QA, a WikiText-trained probe with no task-specific tuning catches about one in eight confident errors that output-confidence monitoring misses, at a 20% flag rate. These results establish signal engineering as a training-time design axis alongside loss and capability. Architecture sets the conditions for observability, and training determines what remains readable.

Transformer可观测性模型监控

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。