用稀疏自编码器解析文生图模型内部机制,一步训练即可跨模型通用。
One-Step is Enough: Sparse Autoencoders for Text-to-Image Diffusion Models
- 在单步SDXL Turbo的更新中训练稀疏自编码器,提取可解释特征。
- 模型特征可控制图像生成,且在4步和基础版模型上仍有效。
- 首个文生图模型的可解释性分析,适合研究生成机制的学者。
对于大语言模型,稀疏自编码器(SAEs)能将难以解读的中间表征分解为可解释特征的稀疏组合,提升可控性与分析能力。然而,文生图模型尚缺乏类似研究。本文首次在单步设置下的SDXL Turbo中训练SAEs,分析其变压器块在去噪U-Net中的更新。结果发现,这些SAEs不仅能泛化到4步的SDXL Turbo,甚至适用于多步的SDXL base模型(不同模型),无需重新训练。我们通过构建基于表征的图像编辑基准RIEBench,验证了所学特征具有可解释性、因果影响性,并揭示了各模块间的功能分化。该工作为理解与操控文生图模型内部机制提供了新路径。
原文摘要 · Abstract (English)
For large language models (LLMs), sparse autoencoders (SAEs) have been shown to decompose intermediate representations that often are not interpretable directly into sparse sums of interpretable features, facilitating better control and subsequent analysis. However, similar analyses and approaches have been lacking for text-to-image models. We investigate the possibility of using SAEs to learn interpretable features for SDXL Turbo, a few-step text-to-image diffusion model. To this end, we train SAEs on the updates performed by transformer blocks within SDXL Turbo's denoising U-net in its 1-step setting. Interestingly, we find that they generalize to 4-step SDXL Turbo and even to the multi-step SDXL base model (i.e., a different model) without additional training. In addition, we show that their learned features are interpretable, causally influence the generation process, and reveal specialization among the blocks. We do so by creating RIEBench, a representation-based image editing benchmark, for editing images while they are generated by turning on and off individual SAE features. This allows us to track which transformer blocks' features are the most impactful depending on the edit category. Our work is the first investigation of SAEs for interpretability in text-to-image diffusion models and our results establish SAEs as a promising approach for understanding and manipulating the internal mechanisms of text-to-image models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。