发现低分辨率训练可保留原梯度信号，大幅缩短扩散模型训练时间。
When does training on downscaled images yield the same gradients?
- 提出梯度变化的双项分解模型，揭示降采样对梯度影响机制。
- 在1024→768降尺度下，噪声σ∈(0.65,0.95)时梯度误差小于5%。
- 实测验证可使LoRA训练提速14.6%，适合资源受限的生成模型微调。
扩散变换器虽生成效果出色，但其训练成本随分辨率呈超线性增长。现有工作基于频谱假设，认为高噪声下降采样后的潜在表示仍保留大部分有效信号。然而，降采样是否能保持原始训练梯度信号仍不清楚。本文将梯度变化归结为两项：一项依赖噪声、由降采样比决定，高噪声时衰减；另一项为不随σ变化的底限，由目标网格的绝对标记数决定，存在于计算图中且无法通过增加噪声消除。实测（路径，σ）映射验证了该理论，并揭示了频谱视角无法解释的结构：在1024→768路径上，存在一个σ∈(0.65,0.95)的窗口，此时降采样梯度与原梯度差异极小。在该路径与噪声窗口内使用降采样步骤训练LoRA适配器，在固定步数预算下，训练时间减少14.6%，权重空间表现接近原模型。代码已公开于https://github.com/sorryhyun/anima_lora。
原文摘要 · Abstract (English)
Diffusion transformers deliver strong image generation, but their training cost grows superlinearly with resolution. Recent work justifies training or sampling at reduced resolution on a spectral premise: at high noise, a downscaled latent preserves almost the full surviving signal. Whether a downscaled step also preserves the native training gradient signal, however, has remained unresolved. We reduce how that signal changes under downscaling to two terms: a noise-dependent term governed by the downscale ratio, which decays at high noise as the spectral premise predicts, and a σ-independent floor governed by the target grid's absolute token count, carried by the compute graph itself and removed by no noise level. The measured (route, σ) map corroborates the account and uncovers structure the spectral picture cannot express: on the 1024->768 route, a window (0.65 < σ< 0.95), predicted by no spectral criterion at any tolerance, where the downscaled gradient stays within a small margin of the native one. Training LoRA adapters with downscaled steps restricted to the routes and noise windows the map validates reduces training time by 14.6% at a fixed step budget while remaining near-native in weight space. Code is available at https://github.com/sorryhyun/anima_lora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。