研究图像分类器集成在数据变化下的线性连接性，发现小学习率和大批量可缓解影响。
Linear Mode Connectivity under Data Shifts for Deep Ensembles of Image Classifiers
- 将数据偏移视为梯度噪声，通过小学习率和大批量降低其影响。
- 小批量下模型易收敛到不同损失区域，大批量则更可能连接到相似最优解。
- 适合关注模型集成稳定性与训练效率平衡的研究者。
线性模式连接性（LMC）关联深度学习中的多个方面，包括在噪声随机梯度下的训练稳定性、局部极小值（盆地）的平滑性与泛化能力、采样模型间的相似性与功能多样性，以及架构对数据处理的影响。本文实验研究了数据偏移下的LMC现象，识别出缓解其影响的条件。我们将数据偏移视为额外的随机梯度噪声，可通过小学习率和大批次大小来减少。这些参数影响模型是否收敛至同一局部极小值，或收敛至损失景观中平滑性和泛化能力不同的区域。尽管通过LMC采样的模型比收敛至不同盆地的模型更频繁地犯相同错误，但LMC的优势在于在训练效率与更大、更多样化的集成收益之间取得平衡。代码与补充材料见 https://github.com/DLR-KI/LMC。本工作已提交IEEE，版权可能转移，之后版本可能不再公开。
原文摘要 · Abstract (English)
The phenomenon of linear mode connectivity (LMC) links several aspects of deep learning, including training stability under noisy stochastic gradients, the smoothness and generalization of local minima (basins), the similarity and functional diversity of sampled models, and architectural effects on data processing. In this work, we experimentally study LMC under data shifts and identify conditions that mitigate their impact. We interpret data shifts as an additional source of stochastic gradient noise, which can be reduced through small learning rates and large batch sizes. These parameters influence whether models converge to the same local minimum or to regions of the loss landscape with varying smoothness and generalization. Although models sampled via LMC tend to make similar errors more frequently than those converging to different basins, the benefit of LMC lies in balancing training efficiency against the gains achieved from larger, more diverse ensembles. Code and supplementary materials are available at https://github.com/DLR-KI/LMC. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。