arXiv:2607.20914cs.LGcs.NI2026-07

解决联邦学习中参数高效微调的谱不匹配问题,提升模型稳定性和性能。

Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning

论文配图:Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning
图 1 · 摘自论文原文
  • 用共享的低秩基实现精确聚合,减少更新空间错位。
  • 通过核范数收缩抑制客户端高秩成分,降低聚合方差。
  • 自适应分配适配头,适合异构数据下的联邦微调场景。

联邦参数高效微调(PEFT)可在去中心化边缘数据上实现大模型的通信高效适配,但在非独立同分布(non-IID)客户端异构性下仍易失效。在低秩适配(LoRA)中,不同客户端可能学习局部有用但频谱错位的更新子空间,导致聚合方差高、全局迁移效果差。我们提出TRISHUL,一种面向鲁棒联邦PEFT的谱控框架。TRISHUL遵循联邦学习无原始数据共享设置,但不提供正式隐私保障。它使用共享冻结的多头低秩基,实现紧凑核心更新的代数精确聚合;在上传前对小规模核心矩阵应用核范数近端收缩,抑制客户端特有高秩谱分量;并基于预训练层容量,采用凹型水分配预算规则非均匀分配适配头。因收缩仅作用于小核心矩阵,TRISHUL计算开销极小,且无需额外每轮通信。在视觉与语言基准测试中,包括CIFAR-100、SVHN、20 Newsgroups、MRQA和GLUE,使用LLaMA3.2-1B模型,TRISHUL相比联邦LoRA基线显著提升收敛速度、稳定性和最终性能,异构性越强,增益越大。

原文摘要 · Abstract (English)

Federated parameter-efficient fine-tuning (PEFT) enables communication-efficient adaptation of large pretrained models on decentralized edge data, but it remains fragile under non-IID client heterogeneity. In low-rank adaptation (LoRA), different clients may learn locally useful but spectrally misaligned update subspaces, causing high-variance aggregation and poor global transfer. We propose TRISHUL, a spectral-control framework for robust federated PEFT. TRISHUL follows the FL no-raw-data-sharing setting but does not itself provide formal privacy guarantees. TRISHUL uses shared frozen multi-head low-rank bases to obtain algebraically exact aggregation of compact core updates, applies nuclear norm proximal shrinkage to suppress client-specific high-rank spectral components before upload, and allocates adaptation heads non-uniformly across layers using a concave water filling budget rule derived from pretrained layer capacity. Because shrinkage is performed only on small core matrices, TRISHUL adds negligible computation and no extra per-round communication over the underlying multi-head PEFT protocol. Across vision and language benchmarks, including CIFAR-100, SVHN, 20 Newsgroups, MRQA, and GLUE with LLaMA3.2-1B, TRISHUL improves convergence, stability, and final performance over federated LoRA baselines, with greater gains under stronger heterogeneity.

联邦学习参数高效谱控制低秩适配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。