arXiv:2607.25583cs.AI2026-07

在6000万参数模型上,小规模微调与量化如何权衡精度与资源消耗。

How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model

论文配图:How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model
图 1 · 摘自论文原文
  • 固定模型和任务,逐项测试LoRA秩、模块选择和量化精度的单独影响
  • LoRA秩16时仅训练不足1%参数,内存降低31%,精度损失仅11.6个百分点
  • 量化至INT8/NF4后内存降至0.6GB,适合资源受限场景部署

参数高效微调(PEFT)和低比特量化已成为在计算资源有限条件下适配语言模型的标准方法,但其交互作用通常在数十亿参数模型上研究,探索成本高昂。本文针对一个具体且可复现的6000万参数编码器-解码器模型(T5-small)和单表文本转SQL基准(WikiSQL),提出反向问题:每项效率优化实际牺牲多少任务精度?我们在单变量控制下系统评估了(i)LoRA秩r ∈ {2, 4, 8, 16, 32},(ii)适配模块集合,以及(iii)数值精度的影响。报告了任务准确率及训练参数量、峰值显存、推理延迟和吞吐量等系统指标,并将模型适配视为多目标权衡而非单一精度优化。结果表明,当LoRA秩为16时,精度(59.6%精确匹配)仅比全量微调(71.2%)低11.6个百分点,同时训练参数少于1%,峰值显存减少31%;超过该秩无明显增益。QLoRA结合INT8与NF4量化分别达到52.8%和53.2%精度,显存仅需0.60GB,显著降低部署成本。所有代码、配置与日志均已公开以保证完全可复现。

原文摘要 · Abstract (English)

Parameter-efficient fine-tuning (PEFT) and low-bit quantization are now standard tools for adapting language models under tight compute budgets, yet their interaction is most often studied on billion-parameter models where the design space is expensive to explore. We ask a complementary question: on a specific, fully reproducible 60M-parameter encoder-decoder model (T5-small) and a single-table text-to-SQL benchmark (WikiSQL), how much task accuracy does each efficiency knob actually cost? We run a controlled, single-variable study over (i) LoRA rank r in {2, 4, 8, 16, 32}, (ii) the set of adapted modules, and (iii) numerical precision. We report task accuracy alongside system-level metrics including trainable parameters, peak training memory, inference latency, and throughput, and frame adaptation as a constrained trade-off rather than an accuracy-only objective. Our results show that LoRA with r=16 recovers within 11.6 percentage points of full fine-tuning accuracy (59.6% vs. 71.2% exact-match) while training fewer than 1% of parameters and consuming 31% less peak GPU memory. Within this setting, rank beyond r=16 yields no measurable accuracy gain. QLoRA with INT8 and NF4 quantization achieves comparable accuracy (52.8% and 53.2%) at dramatically lower memory cost (0.60 GB each), demonstrating a compelling trade-off for memory-constrained deployments. All code, configurations, and logs are released for full reproducibility.

LoRA量化微调小模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。