通过线性探测推导出修正流最优自蒸馏方法,提升生成模型质量。
Optimal Self-Distillation for Rectified Flow via Linear Probing

- 基于线性探测建立精确仿射路径,推导出最优混合系数公式。
- 在教师模型非平稳时,可严格降低速度风险并改善生成效果。
- 无需网格搜索,支持单次交叉验证,适合快速部署到图像生成等场景。
现代生成模型越来越多地利用模型自身生成的信号进行训练,这既带来自我优化的机会,也存在性能退化的风险。本文研究了修正流(Rectified Flow, RF)的最优自蒸馏(SD):给定一个次优的教师速度场,能否通过学生模型结合真实和教师速度场的混合训练,实现对教师的可证明改进?针对固定插值对上的线性修正流与岭正则化情形,本文证明了一个精确的仿射路径恒等式,推导出最优混合系数的闭式解,并表明只要教师在正则化路径上风险不恒定,就能严格降低整体速度风险。最优系数遵循符号规则:正向混合修正欠正则化教师,负向混合修正过正则化教师。此外,提出一种单次广义交叉验证(GCV)与验证调参方法,避免了混合权重的网格搜索与重复拟合。结合修正流的Wasserstein收敛界,进一步证明最优自蒸馏能改善控制连续时间与有限步生成误差的速度估计项。在高斯模型、高斯混合及图像数据上的实验表明,最优自蒸馏在速度风险、模式恢复和有限步生成方面均优于教师模型和纯蒸馏方法。
原文摘要 · Abstract (English)
Modern generative models are increasingly trained using model-generated signals, creating both opportunities for self-improvement and risks of collapse. We study optimal self-distillation (SD) for rectified flow (RF): given a suboptimal teacher velocity field, can a student trained on a mixture of true RF velocities and teacher velocities provably improve the teacher? For linear RF with ridge regularization on fixed interpolation pairs, we prove an exact affine path identity, derive the optimal mixing coefficient in closed form, and show strict improvement in integrated velocity risk whenever the teacher risk is nonstationary along the regularization path. The optimal coefficient obeys a sign rule: positive mixing corrects under-regularized teachers, while negative mixing corrects over-regularized teachers. We also give one-shot generalized cross-validation (GCV) and validation tuning procedure that avoids grid search over mixing weights and repeated refitting. Combining this theorem with RF Wasserstein convergence bounds, we show that optimal self-distillation improves the velocity estimation terms controlling continuous-time and finite-step generation error. Experiments with Gaussian models, Gaussian mixtures, and image data show that optimal self-distillation improves velocity risk, mode recovery, and finite-step generation relative to both the teacher and pure distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。