arXiv:2608.03605cs.AI2026-08

提出高效压缩方法,让联邦学习中的低秩适配更省带宽且不丢精度。

FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation

论文配图:FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation
图 1 · 摘自论文原文
  • 在坐标空间中重构聚合参数,避免传统平均带来的误差。
  • 通信量降低60%以上,精度接近未压缩基线(如72.3%准确率)。
  • 适合资源受限的边缘设备参与联邦大模型微调。

联邦低秩适配(LoRA)可在不集中私有数据的前提下实现大语言模型的高效协同微调。然而,LoRA的双因子参数化导致客户端间聚合失配:直接平均因子无法恢复其诱导更新的平均。该问题可通过在全权重空间形成精确聚合后重新压缩来避免,但分解所得稠密矩阵计算成本高、内存占用大。本文提出FraQ,一种高效的坐标空间重压缩方法。从精确表示聚合结果的堆叠因子出发,FraQ将其分解为正交基和紧凑坐标矩阵;通过小规模格拉姆矩阵恢复奇异谱,选择满足指定能量阈值的最小秩,并将选定坐标子空间通过基映射回构建全局适配器。在文本分类与常识推理基准测试中,FraQ在显著降低下行通信量的同时,保持接近未压缩基线的性能(如72.3%准确率),且服务器端重压缩开销极低。

原文摘要 · Abstract (English)

Federated fine-tuning with Low-Rank Adaptation (LoRA) enables efficient collaborative adaptation of Large Language Models (LLMs) without centralizing private data. However, LoRA's two-factor parameterization creates an aggregation mismatch across clients: naively averaging the factors does not recover the average of their induced updates. This mismatch can be avoided by forming the exact aggregate in the full weight space and then recompressing it, but decomposing the resulting dense matrix is computationally expensive and memory-intensive. We propose FraQ, an efficient coordinate-space recompression method for federated LoRA. Starting from stacked factors that exactly represent the aggregate, FraQ factorizes it into an orthonormal basis and a compact coordinate matrix. It then recovers the singular spectrum from a small Gram matrix, selects the smallest rank satisfying a prescribed energy threshold, and maps the selected coordinate subspace back through the basis to construct the global adapter. Experiments on text classification and commonsense reasoning benchmarks show that FraQ achieves accuracy close to uncompressed baselines while substantially reducing downlink communication with low server-side recompression overhead.

联邦学习低秩适配通信优化大模型微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。