arXiv:2503.07274cs.LGcs.CV2025-03被引 5

用轻量适配器实现单次前向传播的无分类器引导,提速一倍且不降质。

Efficient Distillation of Classifier-Free Guidance using Adapters

  • 用极小额外参数(约2%)训练适配器模拟无分类器引导。
  • 仅需一半神经网络评估次数(NFE),FID表现媲美甚至超越原版。
  • 可在单张消费级显卡上蒸馏大模型,适合资源有限的研究者。

无分类器引导(CFG)对条件扩散模型至关重要,但会使每次推理步骤的神经网络评估次数(NFE)翻倍。为缓解此效率问题,我们提出适配器引导蒸馏(AGD),通过轻量适配器在单次前向传播中模拟CFG,使采样速度提升一倍,同时保持或改善样本质量。与以往需微调整个模型的蒸馏方法不同,AGD冻结基础模型,仅训练约2%的额外参数,显著降低蒸馏阶段资源开销。该方法保留原始模型权重,适配器可无缝与同源其他检查点组合使用。我们还通过在CFG引导轨迹上训练,解决了现有蒸馏方法训练与推理间的不匹配问题。大量实验表明,AGD在多种架构下以仅一半NFE达到与CFG相当或更优的FID表现。特别地,本方法可在单张24GB显存的消费级GPU上蒸馏约2.6B参数的大模型,较此前需多张高端显卡的方法更具可及性。代码将公开发布。

原文摘要 · Abstract (English)

While classifier-free guidance (CFG) is essential for conditional diffusion models, it doubles the number of neural function evaluations (NFEs) per inference step. To mitigate this inefficiency, we introduce adapter guidance distillation (AGD), a novel approach that simulates CFG in a single forward pass. AGD leverages lightweight adapters to approximate CFG, effectively doubling the sampling speed while maintaining or even improving sample quality. Unlike prior guidance distillation methods that tune the entire model, AGD keeps the base model frozen and only trains minimal additional parameters ($\sim$2%) to significantly reduce the resource requirement of the distillation phase. Additionally, this approach preserves the original model weights and enables the adapters to be seamlessly combined with other checkpoints derived from the same base model. We also address a key mismatch between training and inference in existing guidance distillation methods by training on CFG-guided trajectories instead of standard diffusion trajectories. Through extensive experiments, we show that AGD achieves comparable or superior FID to CFG across multiple architectures with only half the NFEs. Notably, our method enables the distillation of large models ($\sim$2.6B parameters) on a single consumer GPU with 24 GB of VRAM, making it more accessible than previous approaches that require multiple high-end GPUs. We will publicly release the implementation of our method.

扩散模型高效推理模型蒸馏轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。