通过分层LoRA分解,实现多模态大模型的高效个性化联邦学习。
PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs

- 将适配器参数分为全局共享与本地私有两部分,分别处理通用语义与边缘适应。
- 通信开销降低近50%,在多个数据集上准确率提升2.4%~4.8%。
- 适合资源受限场景下的边缘智能系统,尤其适用于需隐私保护的协同学习。
自主智能代理正重塑通信与网络,通过部署能在网络边缘协作学习并保障数据隐私的智能体。在分布式网络环境中,多模态大语言模型(MLLMs)作为边缘设备的认知引擎,但联邦微调面临在异构网络条件下平衡全局知识聚合与本地适应的挑战。传统联邦协议依赖统一参数聚合,混淆了域不变特征与客户端特异性,导致个性化不足且通信开销过大。为此,我们提出PFAdapter,一种通信高效的框架,引入分层LoRA分解,显式分离适配器参数为全局共享与本地私有组件。查询和键投影归于全局同步以捕捉跨网络的通用多模态语义,而值和输出投影保持本地化以实现边缘特定适应。此外,基于Frobenius范数的正交性正则化强制两组件严格分离,避免冗余特征学习。选择性聚合协议仅同步全局共享组件,保留本地专长,通信成本减少近50%。在VQA-RAD、SLAKE、Hateful Memes和CrisisMMD数据集上的大量实验表明,PFAdapter持续优于现有先进基线,各类边缘智能任务中准确率提升2.4%至4.8%。因此,该框架为资源受限通信网络中的智能代理部署提供了高效解决方案。
原文摘要 · Abstract (English)
Agentic AI systems are reshaping communications and networking by deploying autonomous intelligent agents capable of collaborative learning while maintaining data privacy at network edges. Within distributed network environments, Multimodal Large Language Models (MLLMs) serve as cognitive engines for edge devices, yet federated fine-tuning faces substantial challenges in balancing global knowledge aggregation with local adaptation under heterogeneous network conditions. Conventional federated protocols typically rely on uniform parameter aggregation, which conflates domain-invariant features with client-specific nuances, thereby resulting in suboptimal personalization and excessive communication overhead. To address these challenges, we propose PFAdapter, a communication-efficient framework introducing hierarchical LoRA decomposition to explicitly separate adapter parameters into global-shared and local-private components. Query and key projections are assigned to global synchronization for capturing universal multimodal semantics across the network, while value and output projections remain localized for edge-specific adaptation. Additionally, orthogonality regularization based on the Frobenius norm enforces strict separation between these components, preventing redundant feature learning. Selective aggregation protocols synchronize only global-shared components across the federated network, preserving local expertise and reducing communication costs by nearly 50%. Extensive experiments on VQA-RAD, SLAKE, Hateful Memes, and CrisisMMD datasets demonstrate that PFAdapter consistently outperforms state-of-the-art baselines, achieving accuracy improvements ranging from 2.4% to 4.8% across diverse edge intelligence tasks. Consequently, our framework establishes an efficient solution for agentic AI deployment in resource-constrained communication networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。