让边缘设备在隐私保护下高效适配大模型,兼顾低资源与少标注。
Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptation
- 边缘端用无标签低清数据,服务器提供少量带标签高清数据,构建实用半监督联邦学习框架。
- 提出FedMox模型,通过稀疏专家网络和空间路由,实现跨分辨率特征对齐,提升适应性能。
- 在自动驾驶数据集上验证,仅用有限内存就显著提升目标检测效果,适合真实部署场景。
基础模型(FMs)具备强大泛化能力,但需针对下游任务适配,尤其在隐私敏感场景中。由于数据隐私法规限制,云端基础模型无法直接访问私有的边缘数据,制约了其适配能力。联邦学习(FL)提供了隐私友好的替代方案,但现有方法忽视了边缘设备的约束——计算资源有限且标注数据稀缺。为此,我们提出实用半监督联邦学习(PSSFL)框架:边缘设备仅持有无标签、低分辨率数据,而服务器拥有少量带标签、高分辨率数据。在此设定下,我们提出联邦专家混合模型(FedMox),一种新型框架以增强联邦学习中的基础模型适配。FedMox通过稀疏专家混合架构应对计算与分辨率不匹配问题,采用空间路由对齐跨分辨率特征,并引入软混合策略稳定半监督学习过程。以目标检测为例,在真实自动驾驶数据集上的实验表明,FedMox能在边缘设备内存受限条件下有效适配基础模型,显著提升性能。本工作为联邦场景下可扩展、隐私保护的基础模型适配铺平了道路。
原文摘要 · Abstract (English)
Foundation models (FMs) exhibit remarkable generalization but require adaptation to downstream tasks, particularly in privacy-sensitive applications. Due to data privacy regulations, cloud-based FMs cannot directly access private edge data, limiting their adaptation. Federated learning (FL) provides a privacy-aware alternative, but existing FL approaches overlook the constraints imposed by edge devices -- namely, limited computational resources and the scarcity of labeled data. To address these challenges, we introduce Practical Semi-Supervised Federated Learning (PSSFL), where edge devices hold only unlabeled, low-resolution data, while the server has limited labeled, high-resolution data. In this setting, we propose the Federated Mixture of Experts (FedMox), a novel framework that enhances FM adaptation in FL. FedMox tackles computational and resolution mismatch challenges via a sparse Mixture-of-Experts architecture, employing a spatial router to align features across resolutions and a Soft-Mixture strategy to stabilize semi-supervised learning. We take object detection as a case study, and experiments on real-world autonomous driving datasets demonstrate that FedMox effectively adapts FMs under PSSFL, significantly improving performance with constrained memory costs on edge devices. Our work paves the way for scalable and privacy-preserving FM adaptation in federated scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。