用随机权重平均增强数据增强，让模型更适应对称性
Boosting Data Augmentation with Stochastic Weight Averaging
- 用奥恩斯坦-乌伦贝克过程模拟训练轨迹，分析SWA的对称性提升机制
- 在无限宽度下，增广数据上的SWA实现超越单纯SWA性能的对称性增强
- 适用于视觉和图分类任务，尤其适合处理离散与连续对称性的场景
学习任务的对称性已成为现代深度学习设计的重要因素。数据增强是一种简单而有效的将对称性融入通用神经网络的方法。近期研究表明，无限大的深度集成在使用增广数据训练时可实现完美的对称性。然而，由于需多次重复训练过程，该方法成本高昂。本文研究随机权重平均（SWA）作为无需重复训练的替代集成技术。通过将训练末期的随机轨迹近似为奥恩斯坦-乌伦贝克过程，我们证明在无限宽极限下，增广数据上的SWA能提供超越纯SWA性能提升的等变性增强。我们在多个涵盖计算机视觉和图分类的模型上，针对离散与连续对称性进行了广泛的数值实验验证。
原文摘要 · Abstract (English)
The symmetries of a learning task have become an important factor in designing modern deep learning solutions. Data augmentation is a straightforward and effective way of incorporating symmetries into a generic neural network. Recent results show that infinitely large deep ensembles show perfect symmetry when trained on augmented data. However, since training ensembles requires repeating the training process many times, this method is costly. In this work, we study stochastic weight averaging (SWA) as an alternative ensembling technique that does not require repeated training runs. We analyze SWA by approximating the stochastic training trajectory at the end of training with an Ornstein--Uhlenbeck process. We show that in the infinite-width limit, SWA on augmented data provides an equiviariance boost that goes beyond what could be expected from the performance increase due to SWA alone. We verify our results with extensive numerical experiments on numerous models spanning computer vision and graph classification with both discrete and continuous symmetries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。