4-bit量化训练扩散模型,内存减少2.25倍,速度提升2.27倍
FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models

- 采用4位量化+三分支混合架构,稳定支持4比特计算
- 在120亿参数模型上实现与全精度相当的生成质量
- 适合需要高效微调大模型的研究者和部署团队
扩散模型已成为高质量生成建模的主流范式,后训练对适配下游应用至关重要。然而,大型扩散模型的后训练仍受高内存开销和慢训练速度制约,现有参数高效微调方法仅部分缓解此问题。为此,我们提出FourTune,一种基于端到端W4A4G4范式的高效后训练框架。FourTune引入三分支混合管道,在标准LoRA架构中加入冻结数值稳定器,隔离量化敏感异常值,实现原生4比特计算下的稳定训练。同时,采用硬件友好的块级量化和定制融合核,支持高效量化反向传播并降低内存带宽开销。在定制化、强化学习与蒸馏任务中,FourTune达到全精度微调质量。在FLUX.1-dev(12B)上,相比BF16 LoRA,FourTune将内存开销降低2.25倍,端到端训练吞吐量提升2.27倍。
原文摘要 · Abstract (English)
Diffusion models have become a dominant paradigm for high-quality generative modeling, while post-training is essential for adapting them to diverse downstream applications. However, post-training of large diffusion models is still challenging due to the prohibitive memory footprints and slow training speed, which existing parameter-efficient fine-tuning methods only partially address. To overcome these limitations, we propose FourTune, an efficient post-training framework for diffusion models based on an end-to-end W4A4G4 paradigm. FourTune introduces a triple-branch hybrid pipeline that augments the standard LoRA architecture with a frozen numerical stabilizer to isolate quantization-sensitive outliers, enabling stable training under native 4-bit computation. In addition, FourTune employs hardware-efficient block-wise quantization and customized fused kernels to support efficient quantized backpropagation and reduce memory bandwidth overhead. Across customization, reinforcement learning, and distillation tasks, FourTune matches the quality of full-precision fine-tuning. On FLUX.1-dev (12B), FourTune reduces memory overhead by 2.25$\times$ and increases end-to-end training throughput by 2.27$\times$ compared to BF16 LoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。