交叉熵是预测编码模型中关键成分,直接影响输出表征质量。
Cross-Entropy Is Load-Bearing: A Pre-Registered Scope Test of the K-Way Energy Probe on Bidirectional Predictive Coding
- 用MSE替代交叉熵训练,探针与Softmax差距缩小一半
- 双向预测编码使探针超越Softmax,证明交叉熵非唯一因素
- 交叉熵主要通过放大输出逻辑值尺度起作用,占比约66%
Cacioli(2026)发现标准判别式预测编码网络上的K-way能量探针近似为log-softmax间隔的单调函数,该结论依赖五个假设,包括输出层使用交叉熵(CE)和前馈推理动态。本预注册研究测试移除交叉熵的影响:分别采用MSE替代CE的标准预测编码(PC),以及双向预测编码(bPC;Oliviers, Tang & Bogacz, 2025)。在CIFAR-10上使用匹配的210万参数骨干网络,共10个随机种子,结果如下:标准PC条件下探针低于Softmax(Delta = -0.082,p < 10^-6),复现了负结果;而bPC条件下探针始终高于Softmax(Delta = +0.008,p = 0.000027),尽管预注册检查显示其潜在激活移动量仅增加1.6倍(阈值10),未显著提升。单独移除交叉熵但保持推断结构,探针与Softmax差距减半(Delta_MSE = -0.037 vs Delta_stdPC = -0.082)。CE训练使输出逻辑值范数约为MSE或bPC训练的15倍。后验温度缩放消融分析表明,探针-Softmax差距中约66%源于可由温度重标定消除的逻辑值尺度效应,约34%反映交叉熵训练表征在尺度不变排名上的优势。文中“元认知”操作性定义为读出对自身一阶正确性的二级判别,不暗示人类自我意识。
原文摘要 · Abstract (English)
Cacioli (2026) showed that the K-way energy probe on standard discriminative predictive coding networks reduces approximately to a monotone function of the log-softmax margin. The reduction rests on five assumptions, including cross-entropy (CE) at the output and effectively feedforward inference dynamics. This pre-registered study tests the reduction's sensitivity to CE removal using two conditions: standard PC trained with MSE instead of CE, and bidirectional PC (bPC; Oliviers, Tang & Bogacz, 2025). Across 10 seeds on CIFAR-10 with a matched 2.1M-parameter backbone, we find three results. The negative result replicates on standard PC: the probe sits below softmax (Delta = -0.082, p < 10^-6). On bPC the probe exceeds softmax across all 10 seeds (Delta = +0.008, p = 0.000027), though a pre-registered manipulation check shows that bPC does not produce materially greater latent movement than standard PC at this scale (ratio 1.6, threshold 10). Removing CE alone without changing inference dynamics halves the probe-softmax gap (Delta_MSE = -0.037 vs Delta_stdPC = -0.082). CE is a major empirically load-bearing component of the decomposition at this scale. CE training produces output logit norms approximately 15x larger than MSE or bPC training. A post-hoc temperature scaling ablation decomposes the probe-softmax gap into two components: approximately 66% is attributable to logit-scale effects removable by temperature rescaling, and approximately 34% reflects a scale-invariant ranking advantage of CE-trained representations. We use "metacognitive" operationally to denote Type-2 discrimination of a readout over its own Type-1 correctness, not to imply human-like introspective access.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。