arXiv:2511.16423cs.AIcs.CL2025-11AAAI被引 1

无需训练,单轮通信即可实现视觉语言模型的高效联邦自适应。

TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Models

  • 利用视觉与文本双通道提取任务相关特征,不依赖训练。
  • 通过贝叶斯原型与全局文本提示对齐,提升跨客户端鲁棒性。
  • 适合资源受限场景,尤其适用于数据异构严重的联邦学习应用。

在联邦学习中,高效轻量地将预训练视觉语言模型(VLMs)适配到下游任务是新兴研究方向。现有方法多需迭代训练,通信开销大且易受攻击。受单轮联邦训练启发,本文提出无需训练的单次联邦自适应框架TOFA,以缓解上述问题。针对当前方法在融合多模态信息、处理严重数据异构及额外训练资源依赖方面的不足,TOFA设计双通道机制:视觉端采用分层贝叶斯模型学习个性化类别原型分布;文本端评估并全局对齐本地生成的文本提示,增强鲁棒性。引入自适应加权校准机制,融合双模态预测,在个性化与鲁棒性间取得平衡。方法完全无需客户端或服务器侧额外训练。在9个不同联邦设置下的数据集上实验表明,TOFA在保持低通信成本的同时显著提升性能。

原文摘要 · Abstract (English)

Efficient and lightweight adaptation of pre-trained Vision-Language Models (VLMs) to downstream tasks through collaborative interactions between local clients and a central server is a rapidly emerging research topic in federated learning. Existing adaptation algorithms are typically trained iteratively, which incur significant communication costs and increase the susceptibility to potential attacks. Motivated by the one-shot federated training techniques that reduce client-server exchanges to a single round, developing a lightweight one-shot federated VLM adaptation method to alleviate these issues is particularly attractive. However, current one-shot approaches face certain challenges in adapting VLMs within federated settings: (1) insufficient exploitation of the rich multimodal information inherent in VLMs; (2) lack of specialized adaptation strategies to systematically handle the severe data heterogeneity; and (3) requiring additional training resource of clients or server. To bridge these gaps, we propose a novel Training-free One-shot Federated Adaptation framework for VLMs, named TOFA. To fully leverage the generalizable multimodal features in pre-trained VLMs, TOFA employs both visual and textual pipelines to extract task-relevant representations. In the visual pipeline, a hierarchical Bayesian model learns personalized, class-specific prototype distributions. For the textual pipeline, TOFA evaluates and globally aligns the generated local text prompts for robustness. An adaptive weight calibration mechanism is also introduced to combine predictions from both modalities, balancing personalization and robustness to handle data heterogeneity. Our method is training-free, not relying on additional training resources on either the client or server side. Extensive experiments across 9 datasets in various federated settings demonstrate the effectiveness of the proposed TOFA method.

联邦学习视觉语言模型无训练多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。