对比分析分布式自监督学习对异构数据的鲁棒性，发现掩码图像建模更稳定。
Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data
- 从理论上证明掩码图像建模比对比学习更抗数据异构
- 网络平均连通度越高，去中心化自监督学习越鲁棒
- 提出MAR损失，提升局部到全局对齐效果，适合实际部署
近期研究提出了分布式自监督学习（D-SSL）方法，以利用海量未标注的分布式数据。然而，D-SSL面临数据异构性的严峻挑战，且现有理论对其响应机制理解有限。为此，本文对非独立同分布（non-IID）环境下D-SSL框架的鲁棒性进行了严格理论分析。结果表明，使用掩码图像建模（MIM）进行预训练比对比学习（CL）更具内在鲁棒性；同时，去中心化自监督学习的鲁棒性随平均网络连通度增加而提升，意味着联邦学习（FL）不劣于去中心化学习（DecL）。这些发现为未来D-SSL算法设计提供了坚实的理论基础。为进一步验证理论，本文提出MAR损失，即在MIM目标中引入局部到全局对齐正则化。在多种模型架构和分布式设置下的大量实验验证了理论结论，并确认了MAR损失的有效性。
原文摘要 · Abstract (English)
Recent research has introduced distributed self-supervised learning (D-SSL) approaches to leverage vast amounts of unlabeled decentralized data. However, D-SSL faces the critical challenge of data heterogeneity, and there is limited theoretical understanding of how different D-SSL frameworks respond to this challenge. To fill this gap, we present a rigorous theoretical analysis of the robustness of D-SSL frameworks under non-IID (non-independent and identically distributed) settings. Our results show that pre-training with Masked Image Modeling (MIM) is inherently more robust to heterogeneous data than Contrastive Learning (CL), and that the robustness of decentralized SSL increases with average network connectivity, implying that federated learning (FL) is no less robust than decentralized learning (DecL). These findings provide a solid theoretical foundation for guiding the design of future D-SSL algorithms. To further illustrate the practical implications of our theory, we introduce MAR loss, a refinement of the MIM objective with local-to-global alignment regularization. Extensive experiments across model architectures and distributed settings validate our theoretical insights, and additionally confirm the effectiveness of MAR loss as an application of our analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。