arXiv:2505.03802cs.LGcs.AI2025-05ACL被引 7

通过联合优化量化位宽与适配器规模,实现低内存下的高性能微调。

Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchies

  • 按语言层次统一优化每层量化位宽与LoRA秩
  • 4比特预算下性能媲美16比特基线模型
  • 适合边缘设备上资源受限的LLM微调

在资源受限的边缘设备上部署和微调大语言模型,需在内存占用与任务性能间权衡。尽管感知量化微调已成可行方案,但现有方法通常将量化与适配器优化分离。我们识别出一个根本性理论约束——保真度-可塑性权衡:某层适应新任务的能力(可塑性)受限于其冻结权重的信息容量(保真度)。对语义关键层过度量化会形成信息瓶颈,任意外部适配器也无法弥补;而对鲁棒句法层保持高精度则浪费内存。为此,我们提出QR-Adaptor框架,联合优化每层量化位宽与LoRA秩。通过将资源分配建模为与语言层次对齐的多目标搜索,系统性地从冗余层释放内存并注入容量关键层。大量实验表明,QR-Adaptor建立了新的帕累托前沿:在严格4比特内存预算下,微调模型性能可媲美16比特基线,证明精准资源对齐与模型规模同等重要。

原文摘要 · Abstract (English)

Deploying and fine-tuning Large Language Models (LLMs) on resource-constrained edge devices requires navigating a strict trade-off between memory footprint and task performance. While Quantization-Aware Fine-tuning has emerged as a viable solution, existing paradigms typically decouple quantization and adapter optimization. This separation overlooks a fundamental theoretical constraint we identify as the \textit{Fidelity-Plasticity Trade-off}: a layer's capacity to adapt to new tasks (Plasticity) is inherently constrained by the information capacity of its frozen weights (Fidelity). Aggressively quantizing semantically critical layers creates an information bottleneck that no amount of adapter rank can recover, while high precision in robust syntactic layers wastes valuable memory. To address this, we introduce \textbf{QR-Adaptor}, a unified framework that jointly optimizes per-layer quantization bit-width and LoRA rank. By formulating resource allocation as a multi-objective search aligned with the model's linguistic hierarchy, our method systematically liberates memory from redundancy-heavy layers to reinvest in capacity-critical ones. Extensive experiments demonstrate that QR-Adaptor establishes a new Pareto frontier: notably, a model fine-tuned under a strict 4-bit memory budget achieves performance rivaling 16-bit baselines, demonstrating that precise resource alignment is as critical as model size.

量化LoRA边缘计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。