提出新评估方法,揭露反馈对齐训练中的隐藏失败模式。
What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility

- 用三个诊断检查替代传统准确率与梯度相似度评估
- 发现两种沉默失效模式:参考梯度崩溃与梯度聚合掩盖层间差异
- 建议分层报告梯度相似度,帮助设计真正有效的反馈对齐方法
尽管深度学习取得成功,但实现生物合理且硬件高效的深层网络训练仍是开放挑战。反馈对齐(FA)通过用固定随机矩阵替代反向传播的对称权重来解决此问题,但其有效性依赖于能否准确评估。当前评估主要依赖任务准确率和信用信号与标准反向传播梯度之间的余弦相似度。我们指出这一评估组合不足,因存在两种独立的隐性失效模式:(1)测量退化,即在末尾层归一化残差架构中,反向传播参考梯度坍缩至数值下限,导致余弦无法解释;(2)聚合坍缩,即整体余弦值掩盖了各层异质性,导致信用集中在网络一端。为此,我们提出基于三重检查——尺度稳定性、参考有效性、深度效用——的新诊断评估协议,并采用逐层而非聚合的余弦报告。在多个架构和方法中,标准评估在所有被审查案例中均无失效信号,而我们的协议能以宽裕校准范围识别全部失败。两种失效模式因果独立:每块尺度惩罚可缓解模式1(残差尺度爆炸引发参考坍缩),但不影响模式2(余弦排序与所有功能指标矛盾)。识别这些隐形失败可防止研究者建立在无效信用分配基础上,为开发真正训练深层的反馈对齐方法提供行动指引。
原文摘要 · Abstract (English)
Despite the success of deep learning, training deep networks in biologically plausible and hardware-efficient ways remains an open challenge. Feedback alignment (FA) methods address this by replacing backpropagation's symmetric backward weights with fixed random matrices, but their effectiveness depends critically on whether they can be accurately evaluated. The standard evaluation relies on two quantities: task accuracy and cosine similarity between the method's credit signal and the backpropagation gradient. We show that this reporting pair is insufficient by identifying two independent failure modes, both silent under current reporting: (1) measurement degeneracy, where the BP reference gradient collapses to the numerical floor in terminal-LayerNorm residual architectures, rendering cosine uninterpretable; and (2) aggregation collapse, where the aggregate cosine masks layerwise heterogeneity that concentrates credit at one end of the network. To address these limitations, we propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting. Across multiple architectures and methods, the standard reporting pair gives no signal of failure in any audited case, while our protocol identifies all failures with wide calibration margins. The two failure modes are causally independent: a per-block scale penalty alleviates Mode 1 (residual scale explosion driving reference collapse) without affecting Mode 2 (cosine ranking that contradicts every functional metric we measured). Identifying these silent failures prevents researchers from building on non-functional credit assignment and provides actionable guidance for developing FA methods that genuinely train deep layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。