arXiv:2510.08023cs.LG2025-10中稿 · the Fourteenth Int…被引 2

模型宽度足以实现线性模式连通,无需复杂排列搜索。

Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivity

  • 仅通过加宽模型并调节软最大化温度即可实现线性路径连接。
  • 在无排列情况下,32倍宽度的ResNet-20仍可达成低损耗线性路径。
  • 适用于研究模型优化路径与结构泛化性的研究人员。

近期研究表明,两个独立训练的模型可通过参数排列实现低损耗线性路径连接(线性模式连通性,LMC)。传统观点认为需足够宽的模型(如ResNet-20使用32×宽度倍数)和有效排列搜索。本文首次发现:即使不进行任何排列,仅通过加宽模型并采用合适的软最大化温度校准,即可实现LMC。我们提出分层指数加权连通性(LEWC)机制,表明合并模型各层输出为原始模型对应层输出的指数加权和,从而等价于原模型的集成输出,促进线性连通。该现象首次揭示模型宽度不仅能增强非线性连通性,还能显著提升线性连通可能性。

原文摘要 · Abstract (English)

Recently, Ainsworth et al. empirically demonstrated that, given two independently trained models, applying a parameter permutation that preserves the input-output behavior allows the two models to be connected by a low-loss linear path. When such a path exists, the models are said to achieve linear mode connectivity (LMC). Prior studies, including Ainsworth et al.(2023), have reported that achieving LMC requires not only an appropriate permutation search but also sufficiently wide models (e.g., a 32 $\times$ width multiplier for ResNet-20). This is broadly believed to be because increasing the model width ensures a large enough space of candidate permutations, increasing the chance of finding one that yields LMC. In this work, we empirically demonstrate that, even without any permutations, simply widening the models is sufficient for achieving LMC when using a suitable softmax temperature calibration. We further explain why this phenomenon arises by analyzing intermediate layer outputs. Specifically, we introduce layerwise exponentially weighted connectivity (LEWC), which states that the output of each layer of the merged model can be represented as an exponentially weighted sum of the outputs of the corresponding layers of the original models. Consequently the merged model's output matches that of an ensemble of the original models, facilitating LMC. To the best of our knowledge, this work is the first to show that widening the model not only facilitates nonlinear mode connectivity, as suggested in prior research, but also significantly increases the possibility of achieving linear mode connectivity.

模型连通性宽度影响线性路径深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。