IntLoRA让量化扩散模型训练更高效，无需再量化就能直接部署。
IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
- 用整数型低秩参数微调量化模型，训练时保持权重量化
- 训练和推理速度大幅提升，低比特下性能不下降
- 适合在消费级显卡上微调量化扩散模型的研究者
在资源受限条件下微调预训练扩散模型已取得显著进展。近期方法通过直接对量化权重使用低秩适配（LoRA）进一步降低训练成本。然而，现有方案在推理阶段效率不足：部署时仍需对微调后的权重进行额外的后训练量化（PTQ），导致低比特下性能明显下降。为此，我们提出IntLoRA，一种对量化扩散模型进行整数型低秩适配的新方法，在微调阶段即考虑推理效率。IntLoRA使预训练权重在整个训练过程中保持量化状态，支持在消费级GPU上高效微调。推理时，可将适配参数无缝合并至原始权重，直接获得量化后的下游权重，无需额外的PTQ步骤。大量实验表明，IntLoRA在训练与推理阶段均实现显著加速，且性能无损失。
原文摘要 · Abstract (English)
Fine-tuning pre-trained diffusion models under limited budgets has gained great success. In particular, the recent advances that directly fine-tune the quantized weights using Low-rank Adaptation (LoRA) further reduces training costs. Despite these progress, we point out that existing adaptation recipes are not inference-efficient. Specifically, additional post-training quantization (PTQ) on tuned weights is needed during deployment, which results in noticeable performance drop when the bit-width is low. Based on this observation, we introduce IntLoRA, which adapts quantized diffusion models with integer-type low-rank parameters, to include inference efficiency during tuning. Specifically, IntLoRA enables pre-trained weights to remain quantized during training, facilitating fine-tuning on consumer-level GPUs. During inference, IntLoRA weights can be seamlessly merged into pre-trained weights to directly obtain quantized downstream weights without PTQ. Extensive experiments show our IntLoRA achieves significant speedup on both training and inference without losing performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。