PairFlow让离散流模型几步生成,无需预训练教师模型。
PairFlow: Closed-Form Source-Target Coupling for Few-Step Generation in Discrete Flow Models
- 通过闭式反演构建源-目标数据对,轻量预处理加速采样。
- 仅需1.7%训练计算量,性能媲美甚至超越两阶段微调方法。
- 提升蒸馏基础模型质量,适合追求高效生成的科研与工程应用。
我们提出$ exttt{PairFlow}$,一种针对离散流模型(DFM)的轻量级预处理方法,实现无需预训练教师模型的少步采样。尽管DFM在离散数据生成上表现优异,但其迭代特性导致采样缓慢。现有加速方法多依赖微调,带来大量额外训练开销。$ exttt{PairFlow}$受ReFlow启发,通过闭式反演直接构建源-目标分布配对样本,无需教师模型。该方法仅需全模型训练1.7%的计算量,却能达到或超越两阶段微调的性能。此外,使用该框架训练的模型可作为更优的蒸馏基底,进一步提升微调后的生成速度。在分子数据、二值图像和RGB图像上的实验验证了方法的广泛适用性与有效性。
原文摘要 · Abstract (English)
We introduce $\texttt{PairFlow}$, a lightweight preprocessing step for training Discrete Flow Models (DFMs) to achieve few-step sampling without requiring a pretrained teacher. DFMs have recently emerged as a new class of generative models for discrete data, offering strong performance. However, they suffer from slow sampling due to their iterative nature. Existing acceleration methods largely depend on finetuning, which introduces substantial additional training overhead. $\texttt{PairFlow}$ addresses this issue with a lightweight preprocessing step. Inspired by ReFlow and its extension to DFMs, we train DFMs from coupled samples of source and target distributions, without requiring any pretrained teacher. At the core of our approach is a closed-form inversion for DFMs, which allows efficient construction of paired source-target samples. Despite its extremely low cost, taking only up to 1.7% of the compute needed for full model training, $\texttt{PairFlow}$ matches or even surpasses the performance of two-stage training involving finetuning. Furthermore, models trained with our framework provide stronger base models for subsequent distillation, yielding further acceleration after finetuning. Experiments on molecular data as well as binary and RGB images demonstrate the broad applicability and effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。