让不同算力的物联网设备协同训练神经网络,自适应切分模型层次。
Hetero-SplitEE: Split Learning of Neural Networks with Early Exits for Heterogeneous IoT Devices
- 各设备按自身算力选择不同切分层,实现异构协作。
- 在CIFAR数据集上保持与标准模型相当的准确率。
- 适合资源差异大的真实物联网场景部署。
深度神经网络的持续扩展已深刻改变机器学习,更大模型在各类任务中表现更优,但其训练所需计算资源急剧增加。为此,联邦学习与分割学习等分布式方法成为可扩展部署的关键。然而,现有分割学习假设客户端同质且使用统一切分点,难以适用于真实物联网系统中算力异构的设备。本文提出Hetero-SplitEE,一种支持异构物联网设备并行协作训练共享深度神经网络的新方法。通过引入异构早期退出机制,在分层训练中允许各客户端根据自身算力自主选择不同的切分层(剪切层)。同时,设计两种协作训练策略:顺序策略通过共享服务器模型依次训练客户端以降低开销;平均策略则支持客户端并行训练,并定期进行跨层聚合。在CIFAR-10、CIFAR-100和STL-10数据集上使用ResNet-18的大量实验表明,该方法在保持竞争性准确率的同时,高效支持多样化的计算约束,为异构物联网生态中的协同深度学习提供了实用解决方案。
原文摘要 · Abstract (English)
The continuous scaling of deep neural networks has fundamentally transformed machine learning, with larger models demonstrating improved performance across diverse tasks. This growth in model size has dramatically increased the computational resources required for the training process. Consequently, distributed approaches, such as Federated Learning and Split Learning, have become essential paradigms for scalable deployment. However, existing Split Learning approaches assume client homogeneity and uniform split points across all participants. This critically limits their applicability to real-world IoT systems where devices exhibit heterogeneity in computational resources. To address this limitation, this paper proposes Hetero-SplitEE, a novel method that enables heterogeneous IoT devices to train a shared deep neural network in parallel collaboratively. By integrating heterogeneous early exits into hierarchical training, our approach allows each client to select distinct split points (cut layers) tailored to its computational capacity. In addition, we propose two cooperative training strategies, the Sequential strategy and the Averaging strategy, to facilitate this collaboration among clients with different split points. The Sequential strategy trains clients sequentially with a shared server model to reduce computational overhead. The Averaging strategy enables parallel client training with periodic cross-layer aggregation. Extensive experiments on CIFAR-10, CIFAR-100, and STL-10 datasets using ResNet-18 demonstrate that our method maintains competitive accuracy while efficiently supporting diverse computational constraints, enabling practical deployment of collaborative deep learning in heterogeneous IoT ecosystems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。