提出SpecTraL方法，实现联邦ViT中低秩适配器的高效全局秩发现。
Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers
- 通过谱变换在低秩空间直接处理客户端适配器，避免密集重构和额外训练。
- 基于随机矩阵理论解析分离共识信号与非独立同分布噪声，自动发现最优层级秩。
- 支持客户端残差维度增量更新，无需重合并基模型，加速收敛且降低通信开销。
在联邦学习中微调视觉变压器（ViTs）时，使用低秩适配器（LoRA）可提升通信效率，但现有聚合策略存在根本性局限。独立平均各LoRA因子在数学上不一致，引入交叉项聚合误差；而通过拼接本地适配器保留异构秩的方案则大幅增加下载成本，并常需将全局更新合并至预训练权重，导致重新初始化延迟和收敛不稳定。其他方法进一步增加服务器端计算负担，如重建稠密权重更新或训练辅助模型以优化聚合误差。本文提出SpecTraL——一种分层全局秩发现的谱变换方法，统一解决上述问题。SpecTraL将客户端的本地LoRA模块堆叠后，在低秩潜在空间直接进行正交Householder变换，避免全局更新的密集重构及服务器端的辅助优化。借助随机矩阵理论中的尖峰协方差模型，SpecTraL可解析分离全局共识信号与非独立同分布噪声，无需人工调参即可发现最优层级全局秩。为匹配后续轮次的本地秩，引入感知填充的初始化框架，使客户端可在不重新合并基模型的前提下融入残差LoRA维度。在DomainNet和NICO++数据集上对ViT-B/16和ViT-L/16进行联邦微调实验表明，该方法显著改善准确率-通信权衡，降低服务器计算量，并消除秩选择的超参数搜索。
原文摘要 · Abstract (English)
Fine-tuning Vision Transformers (ViTs) with low-rank adapters (LoRA) promises better communication efficiency under federated setup, yet existing aggregation strategies face fundamental limitations. Independently averaging these LoRA factors is mathematically inconsistent, introducing cross-term aggregation error. In contrast, approaches that preserve heterogeneous client ranks by concatenating local adapters on the server substantially increase download cost and often require merging global LoRA updates into pretrained weights on the clients, causing reinitialization lag and unstable convergence. Other approaches further increase server-side overhead by reconstructing dense weight updates or training auxiliary models to refine aggregation error. In this work, we propose SpecTraL, spectral transformation for layer-wise global rank discovery, that resolves these challenges within a unified design. SpecTraL stacks local LoRA modules from clients and performs orthonormal Householder Transformation of the stacked adapters directly in the low-rank latent space, eliminating dense reconstruction of the global update and any auxiliary refinement on the server. By leveraging the Spiked Covariance Model from Random Matrix Theory, SpecTraL analytically separates the global consensus signal from non-IID noise, discovering optimal layer-wise global ranks without manual hyperparameter tuning. To match local ranks in subsequent rounds, we introduce a padding-aware initialization framework that lets clients incorporate residual LoRA dimensions without re-merging them into the pre-trained base model. Experiments on federated fine-tuning of ViT-B/16 and ViT-L/16 over DomainNet and NICO++ demonstrate improved accuracy-communication trade-offs, reduced server computation, and elimination of hyperparameter search for rank selection. Our code is available at https://github.com/DASS-Lab-Group/SpecTraL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。