用模型自身中间层自对齐，提升生成质量与训练效率
LayerSync: Self-aligning Intermediate Layers
- 用模型内部各层表示自监督对齐，无需外部指导
- 图像生成训练提速8.75倍，质量提升23.6%
- 通用性强，适用于音频、视频、动作生成等多模态
我们提出 LayerSync，一种无需领域先验的扩散模型优化方法，可提升生成质量和训练效率。已有研究指出生成质量与模型中间表示密切相关，外部引导能加速训练。本文重新构想该范式，通过模型自身中间表示进行正则化。基于不同层表示质量差异的观察，我们发现语义最丰富的层可作为弱层的内在引导，减少对外部监督的依赖。LayerSync 是一个自洽、即插即用的正则化项，训练无额外开销，且可泛化至视觉以外的模态。该方法无需预训练模型或额外数据。我们在图像生成上广泛验证，并拓展至音频、视频和运动生成。结果表明，其持续提升生成质量与训练效率：例如在 ImageNet 上使基于流的 Transformer 训练速度提升 8.75 倍，生成质量提升 23.6%。代码已开源。
原文摘要 · Abstract (English)
We propose LayerSync, a domain-agnostic approach for improving the generation quality and the training efficiency of diffusion models. Prior studies have highlighted the connection between the quality of generation and the representations learned by diffusion models, showing that external guidance on model intermediate representations accelerates training. We reconceptualize this paradigm by regularizing diffusion models with their own intermediate representations. Building on the observation that representation quality varies across diffusion model layers, we show that the most semantically rich representations can act as an intrinsic guidance for weaker ones, reducing the need for external supervision. Our approach, LayerSync, is a self-sufficient, plug-and-play regularizer term with no overhead on diffusion model training and generalizes beyond the visual domain to other modalities. LayerSync requires no pretrained models nor additional data. We extensively evaluate the method on image generation and demonstrate its applicability to other domains such as audio, video, and motion generation. We show that it consistently improves the generation quality and the training efficiency. For example, we speed up the training of flow-based transformer by over 8.75x on ImageNet dataset and improved the generation quality by 23.6%. The code is available at https://github.com/vita-epfl/LayerSync.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。