用去相关技术加速视觉Transformer训练，省时省电还提效果
Decorrelation Speeds Up Vision Transformers
- 在MAE预训练中引入去相关反向传播，逐层降低输入相关性以加快收敛
- 在ImageNet-1K上训练时间减少21.1%，碳排放降低21.4%，分割性能提升1.1点mIoU
- 适合资源受限工业场景，真实数据验证有效，兼顾效率与性能
视觉变换器（ViTs）的掩码自编码器（MAE）预训练在低标签数据下表现优异，但计算成本高，在时间和资源受限的工业场景中不实用。本文通过将去相关反向传播（DBP）引入MAE预训练，该优化方法在每层迭代降低输入相关性以加速收敛。仅对编码器应用DBP即可实现更快预训练且不失稳定性。为模拟数据受限场景，我们在ImageNet-1K预训练和ADE20K微调中使用各数据集的随机子集进行评估。结果显示，DBP-MAE将达到基线性能的墙钟时间缩短21.1%，碳排放降低21.4%，分割任务的mIoU提升1.1点。在自有工业数据上的预训练与微调也获得类似收益，证实该方法在真实场景中的适用性。结果表明，DBP可显著降低大规模ViT预训练的时间与能耗，同时提升下游性能。
原文摘要 · Abstract (English)
Masked Autoencoder (MAE) pre-training of vision transformers (ViTs) yields strong performance in low-label data regimes but comes with substantial computational costs, making it impractical in time- and resource-constrained industrial settings. We address this by integrating Decorrelated Backpropagation (DBP) into MAE pre-training, an optimization method that iteratively reduces input correlations at each layer to accelerate convergence. Applied selectively to the encoder, DBP achieves faster pre-training without loss of stability. To mimic constrained-data scenarios, we evaluate our approach on ImageNet-1K pre-training and ADE20K fine-tuning using randomly sampled subsets of each dataset. Under this setting, DBP-MAE reduces wall-clock time to baseline performance by 21.1%, lowers carbon emissions by 21.4%, and improves segmentation mIoU by 1.1 points. We observe similar gains when pre-training and fine-tuning on proprietary industrial data, confirming the method's applicability in real-world scenarios. These results demonstrate that DBP can reduce training time and energy use while improving downstream performance for large-scale ViT pre-training. Keywords: Deep learning, Vision transformers, Efficient AI, Decorrelation
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。