让边缘设备用小模型高效训练大模型，不传数据也能学新知识。
EFFEKT: Efficient Federated Knowledge Transfer to Foundation Models
- 客户端用轻量代理模型，服务器端用大模型协同学习
- 在多个数据集上性能超越现有方法，客户端计算量极低
- 适合资源受限的边缘设备，保护隐私且能持续更新知识
近年来，数据保护法规推动了联邦学习（FL）在隐私保护去中心化训练中的广泛应用。然而，模型规模扩大给客户端设备带来巨大计算负担，限制了其在资源受限场景的应用。我们提出一种新型多领域联邦学习框架，其中轻量级客户端代理模型与服务端基础模型（FM）协作，在不共享私有数据的前提下学习新概念。我们的方法 EFFEK T 通过创新的双向交叉蒸馏策略，实现服务器端高效训练特定领域的 LoRA 适配器，同时保持基础模型与代理提取器之间的特征空间对齐。在多个真实世界数据集上的实验及在低功耗边缘设备上的部署表明，该方法在多数领域均优于当前最先进基线，且客户端计算开销极低。
原文摘要 · Abstract (English)
Recent data protection laws have accelerated the adoption of Federated Learning (FL) for privacy-preserving decentralized training. Nevertheless, increasing model sizes impose substantial computational demands on client devices, limiting FL applicability in resource-constrained settings. We introduce a novel multi-domain federated learning framework in which lightweight client-side proxy models collaborate with a server-side Foundation Model (FM) to learn new concepts without sharing private data. Our approach, EFFEKT, enables efficient server-side training of domain-specific LoRA adapters while preserving feature-space alignment between the FM and proxy extractors via novel bi-directional cross-distillation strategies. Experiments on multiple real-world datasets and deployments on low-power edge devices demonstrate improvements over state-of-the-art baselines in most considered domains while maintaining lightweight computation at the client side.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。