通过敏感度引导的量化压缩,显著降低边缘设备微调大模型的通信开销。
FED-FSTQ: Fisher-Guided Token Quantization for Communication-Efficient Federated Fine-Tuning of LLMs on Edge Devices
- 用轻量级词元级费雪信息估算语义重要性,指导稀疏化与混合精度量化。
- 相比基线减少46倍上行流量,52%的训练耗时,每轮节省超80%能耗。
- 无需改动现有流程,适合资源受限的移动/边缘设备部署。
联邦微调为在不集中私有数据的前提下适配大语言模型提供了可行路径。然而,在移动端部署中,训练耗时常受异构带宽、间歇参与和非独立同分布客户端数据影响,由慢节点拖累的上行通信成为瓶颈。尽管参数高效微调(PEFT)方法如LoRA和QLoRA减少了本地内存与可训练参数,但适配器更新的重复传输仍是主要障碍。本文提出Fed-FSTQ,一种面向通信高效的联邦LLM微调的语义敏感性感知通信控制机制。该方法使用轻量级词元级费雪代理估计语义敏感度,结合词元引导的稀疏化与混合精度适配器更新量化,将更高通信保真度分配给语义承载关键证据,抑制冗余传输。该方法与标准联邦PEFT流程兼容,无需修改服务器聚合规则。在非独立同分布划分下的多语言问答与医疗问答任务实验表明,Fed-FSTQ相较基准联邦LoRA减少达到固定质量阈值所需的累计上行流量46倍,提升慢节点限制下的时间-准确率效率52%。在修正后的受控LTE-20Mbps场景下,每轮耗时从414.60秒降至67.29秒,能耗由839.20焦耳降至146.28焦耳,提速6.16倍。在NVIDIA Jetson类边缘设备上,费雪引导的词元缩减还带来最高1.55倍的推理加速,验证了其在严苛资源约束下的可部署性。
原文摘要 · Abstract (English)
Federated fine-tuning provides a practical route to adapt large language models (LLMs) on edge devices without centralizing private data. However, in mobile deployments, the training wall-clock is often dominated by straggler-limited uplink communication under heterogeneous bandwidth, intermittent participation, and non-IID client data. Although parameter-efficient fine-tuning (PEFT) methods such as LoRA and QLoRA reduce local memory and trainable parameters, repeated transmission of adapter updates remains a major bottleneck. We propose Fed-FSTQ, a semantic-sensitivity-aware communication-control primitive for communication-efficient federated LLM fine-tuning. Fed-FSTQ uses a lightweight token-level Fisher proxy to estimate semantic sensitivity, couples token-guided sparsification with mixed-precision adapter-update quantization, and allocates higher communication fidelity to semantically load-bearing evidence while suppressing redundant transmission. The method is drop-in compatible with standard federated PEFT pipelines and requires no change to the server aggregation rule. Experiments on multilingual QA and medical QA under non-IID partitions show that Fed-FSTQ reduces cumulative uplink traffic required to reach a fixed quality threshold by 46-fold relative to a Fed-LoRA baseline and improves straggler-limited wall-clock time-to-accuracy by 52%. Under the corrected Controlled LTE-20Mbps accounting, Fed-FSTQ reduces per-round time from 414.60s to 67.29s and reduces per-round energy from 839.20J to 146.28J, yielding a 6.16-fold speedup. On NVIDIA Jetson-class edge devices, Fisher-guided token reduction also yields up to a 1.55-fold inference speedup, demonstrating deployability under tight resource constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。