用中心数据优化分散模型,提升混合数据下的训练效果
Guided Model Merging for Hybrid Data Learning: Leveraging Centralized Data to Refine Decentralized Models
- 构建分散模型的全局结构图谱,用中心数据精炼全局模型
- 在混合数据下比纯中心或纯分散方法性能更优,收敛更快
- 适用于数据异构、噪声干扰等复杂场景,实用性强
当前网络训练主要聚焦于集中式或分布式数据范式,但实际中数据常呈现混合特性:分布式数据虽丰富但存在异质性和通信限制,集中式数据量少且可能不具代表性,但支持高效访问与精细处理。本文提出一种新框架,通过分散模型构建模型图谱,并利用集中式数据在该结构空间中精炼全局模型,再用此模型重新初始化分散模型。该方法融合联邦学习与模型合并,理论上实现比仅依赖分布式数据更快的收敛(因合并过程降低方差)。大量实验表明,本框架在各类混合设置下均优于纯集中式、纯分布式及现有混合适配方法,即使在集中与分散数据分布差异大或含噪声时仍保持鲁棒性,显著拓展适用范围。
原文摘要 · Abstract (English)
Current network training paradigms primarily focus on either centralized or decentralized data regimes. However, in practice, data availability often exhibits a hybrid nature, where both regimes coexist. This hybrid setting presents new opportunities for model training, as the two regimes offer complementary trade-offs: decentralized data is abundant but subject to heterogeneity and communication constraints, while centralized data, though limited in volume and potentially unrepresentative, enables better curation and high-throughput access. Despite its potential, effectively combining these paradigms remains challenging, and few frameworks are tailored to hybrid data regimes. To address this, we propose a novel framework that constructs a model atlas from decentralized models and leverages centralized data to refine a global model within this structured space. The refined model is then used to reinitialize the decentralized models. Our method synergizes federated learning (to exploit decentralized data) and model merging (to utilize centralized data), enabling effective training under hybrid data availability. Theoretically, we show that our approach achieves faster convergence than methods relying solely on decentralized data, due to variance reduction in the merging process. Extensive experiments demonstrate that our framework consistently outperforms purely centralized, purely decentralized, and existing hybrid-adaptable methods. Notably, our method remains robust even when the centralized and decentralized data domains differ or when decentralized data contains noise, significantly broadening its applicability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。