arXiv:2608.23816cs.LG2026-08

AQLoRA无需搜索,快速实现量化LoRA微调,兼顾速度与精度。

AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning

  • 通过一次权重扫描,自适应选择需保持fp16的层以避免解量化开销。
  • 在1.4B到14B模型上提速11.1%,精度损失仅约1点。
  • 适用于追求高效微调且资源有限的开发者,尤其适合多模型部署场景。

量化微调(QLoRA)虽节省显存但训练慢,因其需实时解量化4比特权重。本文提出AQLoRA(自适应量化LoRA),仅需一次CPU遍历权重即可完成配置,无需搜索或校准数据。该方法按NF4重构误差排序各层,于内存预算内保留前K层为fp16,避免解量化开销,从而提升速度。质量设置适配所有层,速度设置仅优化顶层模块,使反向传播提前终止。该策略在秒级内复现Unsloth的手动动态4比特选择,而基于搜索的分配需多次校准。我们在六种模型、四种架构家族(1.4B至14B)上评估,速度设置比调优后的QLoRA快11.1%±2.7%,最慢仍快7%;精度损失约1点。质量设置提速4.8%±2.4%,精度与QLoRA持平,接近fp16 LoRA,仅多占用0.2 GiB内存。误差条来自独立实验会话间差异,而非单次会话内部。测量表明:固定时长优于固定步数,噪声底限应从重复实验获取,而非近似方法;整轮重复至关重要,局部计算低估真实不确定性数倍,随机种子影响极小。我们验证了控制组,发现按权重密度选适配器层或按量化误差选保护层均无效,保护层数量而非具体身份决定加速效果。

原文摘要 · Abstract (English)

Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We present AQLoRA (Adaptive-Quantization LoRA), a recipe that buys part of that time back. One CPU pass over the weights sets everything, with no search and no calibration data. The pass ranks layers by NF4 reconstruction error and keeps the top-K in fp16 under a memory budget. Those layers skip dequantization, which is where the speed comes from. A quality setting adapts every layer. A speed setting adapts only the top blocks, so the backward pass stops early. The rule reproduces Unsloth's hand-curated dynamic-4bit selection exactly, in seconds, where search-based allocation needs repeated calibration passes. We evaluate on Commonsense-170K across six models and four architecture families, from 1.4B to 14B. The speed setting trains 11.1 +/- 2.7% faster than well-tuned QLoRA and gives up about one accuracy point. It was faster in all nine independent timing sessions, at worst by 7%. The quality setting trains 4.8 +/- 2.4% faster. Its accuracy is level with QLoRA on every model and within a point of fp16 LoRA, for 0.2 GiB more memory. These error bars are measured between independent sessions, not within one. Earning them taught us three rules for timing on shared hardware. Fix the measurement duration, not the step count. Measure the noise floor from a duplicated arm, not a nearly identical method. Repeat whole sessions: a floor computed inside one sweep understates the real uncertainty several times over, and the random seed controls almost none of it. We validate the recipe with controls and report the two that failed. Choosing adapter layers by weight density is no better than random. Choosing protected layers by quantization error is not either. The count of protected layers, not their identity, carries the speed effect.

LoRA量化微调加速自适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。