让残差模块学新特征：用正交更新提升深度网络稳定性和精度
Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks
- 将模块输出分解为与输入流正交的部分，只添加新方向信息
- 在ImageNet-1k上使ViT-B准确率提升3.78个百分点，通用性好
- 适合追求高稳定性与泛化能力的深度模型研究者
残差连接对深度神经网络至关重要，能缓解梯度消失问题。但标准残差更新中，模块输出直接加到输入流上，可能导致更新主要强化或调制原有方向，未能充分学习新特征。本文提出正交残差更新：将模块输出相对于输入流进行分解，仅添加与之正交的部分。该设计引导模块主要贡献新的表示方向，促进更丰富的特征学习并提升训练效率。实验表明，该方法在多种架构（ResNetV2、Vision Transformers）和数据集（CIFARs、TinyImageNet、ImageNet-1k）上均显著提升泛化准确率与训练稳定性，例如在ImageNet-1k上使ViT-B的top-1准确率提升3.78个百分点。
原文摘要 · Abstract (English)
Residual connections are pivotal for deep neural networks, enabling greater depth by mitigating vanishing gradients. However, in standard residual updates, the module's output is directly added to the input stream. This can lead to updates that predominantly reinforce or modulate the existing stream direction, potentially underutilizing the module's capacity for learning entirely novel features. In this work, we introduce Orthogonal Residual Update: we decompose the module's output relative to the input stream and add only the component orthogonal to this stream. This design aims to guide modules to contribute primarily new representational directions, fostering richer feature learning while promoting more efficient training. We demonstrate that our orthogonal update strategy improves generalization accuracy and training stability across diverse architectures (ResNetV2, Vision Transformers) and datasets (CIFARs, TinyImageNet, ImageNet-1k), achieving, for instance, a +3.78 pp top-1 accuracy gain for ViT-B on ImageNet-1k.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。