联邦学习下个性化视觉语言模型微调框架，提升异构数据适应能力。
FedVLM: Scalable Personalized Vision-Language Models through Federated Learning
- 采用可个性化调整的LoRA参数，动态适配各客户端数据分布。
- 在RLAIF-V数据集上，本地性能比标准LoRA提升24.5%。
- 适合隐私敏感、数据分布不均的分布式视觉语言任务场景。
视觉语言模型（VLMs）展现出强大的零样本与少样本学习能力，广泛应用于下游任务。然而，在联邦学习环境中，由于数据分散且非独立同分布（non-iid），大规模微调仍具挑战性。现有参数高效微调方法如LoRA虽降低计算开销，但在异构数据下泛化效果不佳。为此，我们提出FedVLM，一种基于联邦LoRA的微调框架，支持去中心化适配视觉语言模型，同时保护模型隐私并减少对集中式训练的依赖。为应对数据异构性，我们引入个性化LoRA（pLoRA），动态调整每个客户端的LoRA参数以匹配其数据分布，显著提升本地适应能力，同时保持全局模型聚合。在RLAIF-V数据集上的实验表明，相比标准LoRA，pLoRA使客户端特定性能提升24.5%，证明了其在非独立同分布设置下的优越适应性。FedVLM为联邦环境下VLM的规模化高效微调提供了可行方案，推动分布式学习中的个性化适配发展。
原文摘要 · Abstract (English)
Vision-language models (VLMs) demonstrate impressive zero-shot and few-shot learning capabilities, making them essential for several downstream tasks. However, fine-tuning these models at scale remains challenging, particularly in federated environments where data is decentralized and non-iid across clients. Existing parameter-efficient tuning methods like LoRA (Low-Rank Adaptation) reduce computational overhead but struggle with heterogeneous client data, leading to suboptimal generalization. To address these challenges, we propose FedVLM, a federated LoRA fine-tuning framework that enables decentralized adaptation of VLMs while preserving model privacy and reducing reliance on centralized training. To further tackle data heterogeneity, we introduce personalized LoRA (pLoRA), which dynamically adapts LoRA parameters to each client's unique data distribution, significantly improving local adaptation while maintaining global model aggregation. Experiments on the RLAIF-V dataset show that pLoRA improves client-specific performance by 24.5% over standard LoRA, demonstrating superior adaptation in non-iid settings. FedVLM provides a scalable and efficient solution for fine-tuning VLMs in federated settings, advancing personalized adaptation in distributed learning scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。