LoRA微调压缩到1.15比特，内存减半且性能损失小。
LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
- 精细量化+动态阈值，实现低于2比特的高效微调
- 最低达1.15比特，4个模型4个数据集均保持高精度
- 适合低资源设备部署，如移动端或边缘计算
随着大语言模型参数量突破百亿，微调成本持续攀升，即使参数高效方法如LoRA也仍需大量资源。本文提出LowRA，首个实现每参数低于2比特的LoRA微调框架，通过优化细粒度量化、映射策略、阈值选择与精度分配，并结合高效的CUDA核函数实现可扩展部署。在4个大模型和4个数据集上的广泛评估显示，LowRA在2比特以上表现优于现有方法，性能稳定降至1.15比特，内存占用最高降低50%。结果表明，超低比特LoRA微调在资源受限场景中具有巨大潜力。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) is increasingly costly as models scale to hundreds of billions of parameters, and even parameter-efficient fine-tuning (PEFT) methods like LoRA remain resource-intensive. We introduce LowRA, the first framework to enable LoRA fine-tuning below 2 bits per parameter with minimal performance loss. LowRA optimizes fine-grained quantization - mapping, threshold selection, and precision assignment - while leveraging efficient CUDA kernels for scalable deployment. Extensive evaluations across 4 LLMs and 4 datasets show that LowRA achieves a superior performance-precision trade-off above 2 bits and remains accurate down to 1.15 bits, reducing memory usage by up to 50%. Our results highlight the potential of ultra-low-bit LoRA fine-tuning for resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。