通过双适配器与剪枝提升联邦多模态学习的效率与泛化能力
Federated Multimodal Learning with Dual Adapters and Selective Pruning for Communication and Computational Efficiency
- 用大小适配器分别实现个性化与高效共享
- 测试准确率更高，客户端性能方差更低，通信计算成本显著下降
- 适合资源受限场景下的分布式多模态模型部署
联邦学习（FL）可在保护数据隐私的前提下实现跨分布式客户端的协同学习。然而，面对异构数据分布时，传统方法常导致全局模型性能不佳且泛化能力弱。本文提出一种新框架，引入双适配器机制：大本地适配器用于客户端个性化，小全局适配器促进高效知识共享。同时，采用剪枝策略，有选择性地移除本地适配器中影响较小的参数，以降低通信开销。在多个视觉与语言任务上的实验表明，该方法在保持高测试准确率的同时，显著降低了客户端间的性能方差和最差表现，并大幅减少通信与计算成本。整体上，有效平衡了模型个性化与泛化之间的权衡，为实际应用中的联邦学习提供了可扩展的解决方案。
原文摘要 · Abstract (English)
Federated Learning (FL) enables collaborative learning across distributed clients while preserving data privacy. However, FL faces significant challenges when dealing with heterogeneous data distributions, which can lead to suboptimal global models that fail to generalize across diverse clients. In this work, we propose a novel framework designed to tackle these challenges by introducing a dual-adapter approach. The method utilizes a larger local adapter for client-specific personalization and a smaller global adapter to facilitate efficient knowledge sharing across clients. Additionally, we incorporate a pruning mechanism to reduce communication overhead by selectively removing less impactful parameters from the local adapter. Through extensive experiments on a range of vision and language tasks, our method demonstrates superior performance compared to existing approaches. It achieves higher test accuracy, lower performance variance among clients, and improved worst-case performance, all while significantly reducing communication and computation costs. Overall, the proposed method addresses the critical trade-off between model personalization and generalization, offering a scalable solution for real-world FL applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。