解决扩散模型多轮编辑中的语义漂移问题,通过低频对齐稳定图像质量。
Why Do DiT Editors Drift? Plug-and-Play Low Frequency Alignment in VAE Latent Space

- 在VAE隐空间中通过低通滤波分离并对齐低频特征,抑制漂移
- 无需重训练,在多轮编辑中显著提升语义一致性与视觉保真度
- 适用于白盒和黑盒模型,即插即用且不依赖扩散参数
最近的扩散变换器(DiT)在单轮图像编辑上表现优异,但多轮编辑常导致语义漂移与质量下降。本文从隐空间频率视角出发,将编辑过程分解为VAE与DiT两部分。系统分析表明,DiT引入主导性的低频漂移,随编辑轮次累积造成语义错位,而VAE仅贡献较稳定的重建偏差。基于此,提出VAE-LFA(低频对齐)方法:在VAE隐空间中通过低通滤波分解各轮差异,将低频统计对齐至前几轮的指数移动平均值,有效抑制累积漂移并保留高频细节。该方法无需重训练、无需真实先验或扩散参数,可适配白盒与黑盒DiT编辑器。白盒场景下可无缝集成,消除冗余的VAE往返;黑盒场景下通过现成的VAE实现跨轮隐空间对齐。大量实验显示,VAE-LFA在多种多轮编辑任务中均显著提升语义一致性和视觉质量,涵盖受控与真实场景图像。
原文摘要 · Abstract (English)
Recent advances in diffusion transformers (DiTs) have enabled promising single-turn image editing capabilities. However, multi-turn editing often leads to progressive semantic drift and quality degradation.In this work, we study this problem from a latent-space frequency perspective by decomposing the editing process into two functional components: VAE and DiT. Through systematic analysis in the VAE latent space, we uncover that the DiT introduces dominant low-frequency drift that accumulates as semantic misalignment across editing rounds, while the VAE contributes comparatively stable reconstruction bias.Based on this insight, we propose VAE-LFA (Low Frequency Alignment), a training-free, plug-and-play method that performs alignment in VAE latent space. VAE-LFA decomposes latent discrepancies across editing rounds via low-pass filtering, and aligns low-frequency statistics to an exponential moving average of previous rounds, effectively suppressing accumulated semantic drift while preserving high-frequency details.Our method requires no retraining, ground-truth priors, or access to diffusion parameters, making it applicable to both white-box and black-box DiT editors. For white-box models, VAE-LFA is seamlessly integrated into the editing pipeline by eliminating redundant VAE round trips; for black-box models, it operates via an off-the-shelf VAE to perform inter-round latent alignment.Extensive experiments demonstrate that VAE-LFA improves semantic consistency and visual fidelity across diverse multi-turn editing scenarios, including both controlled and in-the-wild images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。