分层低秩适配提升联邦学习中视觉模型的个性化与泛化能力
HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learning
- 在根、组、客户端三级部署适配器，分别捕捉全局、子群和个体知识
- 跨层级正交优化使各层级更新独立，提升个性化性能，测试准确率提升3.2%以上
- 适用于有隐含客户结构的联邦学习场景，尤其适合资源受限设备上的模型微调
视觉变换器（ViTs）因强大的迁移能力被广泛应用于视觉任务。在联邦学习（FL）中，全量微调通信开销大，低秩适配（LoRA）提供了高效且通信友好的方式来适配ViTs。然而，现有基于LoRA的联邦调优方法忽略了真实场景中客户端的潜在结构，限制了共享表示学习，并阻碍了对未见客户端的有效适应。为此，我们提出HiLoRA，一种分层低秩适配框架，在根、聚类和叶三级放置适配器，分别捕获全局、子群和客户端特定知识。通过跨层级正交性和级联优化，HiLoRA分离更新子空间，并将每层与各自的个性化目标对齐。特别地，我们设计了LoRA-子空间自适应聚类机制，通过子空间相似性分析推断隐含客户端群体，促进结构对齐客户端之间的知识共享。理论上，我们建立了分层泛化分析支持该设计。在使用CIFAR-100和DomainNet数据集的ViT主干网络上，实验表明其在个性化和泛化方面均有持续提升。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have been widely adopted in vision tasks due to their strong transferability. In Federated Learning (FL), where full fine-tuning is communication heavy, Low-Rank Adaptation (LoRA) provides an efficient and communication-friendly way to adapt ViTs. However, existing LoRA-based federated tuning methods overlook latent client structures in real-world settings, limiting shared representation learning and hindering effective adaptation to unseen clients. To address this, we propose HiLoRA, a hierarchical LoRA framework that places adapters at three levels: root, cluster, and leaf, each designed to capture global, subgroup, and client-specific knowledge, respectively. Through cross-tier orthogonality and cascaded optimization, HiLoRA separates update subspaces and aligns each tier with its residual personalized objective. In particular, we develop a LoRA-Subspace Adaptive Clustering mechanism that infers latent client groups via subspace similarity analysis, thereby facilitating knowledge sharing across structurally aligned clients. Theoretically, we establish a tier-wise generalization analysis that supports HiLoRA's design. Experiments on ViT backbones with CIFAR-100 and DomainNet demonstrate consistent improvements in both personalization and generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。