用自监督原理分析扩散模型的表征空间,发现中间噪声时表征最优。
Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles

- 分解特征为不变与残差成分,提出基于Fisher的污染率指标ICR。
- 中间噪声水平下不变性最强,下游分类性能最好。
- 通过训练特征变化可提前检测模型是否开始记忆数据。
扩散模型展现出强大的生成能力,同时作为自监督表征学习器也表现优异,但这两者之间的联系尚未深入探索。受自监督学习启发,我们提出一个联合评估扩散模型表征与生成能力的框架。具体地,将特征分解为不变和残差成分,推导出基于Fisher的不变污染率(ICR),量化残差变异对表征空间中不变信号的污染程度。利用该框架分析扩散模型的判别与生成行为。在表征方面,发现不变性在中间噪声水平达到峰值,此时下游分类性能最佳。在生成方面,研究了数据有限条件下训练从真实泛化转向记忆的过程,表明ICR可作为敏感的训练期指示器:沿Fisher方向残差能量上升标志着记忆开始,仅从训练特征即可检测,无需外部评估或预留测试集。总体而言,我们的结果表明,可通过其学习表征的几何结构,从自监督视角监控扩散模型。
原文摘要 · Abstract (English)
Diffusion models have demonstrated remarkable generative capabilities and have also emerged as powerful self-supervised representation learners, yet the connection between these two abilities remains less explored. Drawing inspiration from self-supervised learning (SSL), we introduce a framework for jointly evaluating the representation and generation capabilities of diffusion models. Specifically, we decompose features into invariant and residual components and derive the Invariant Contamination Ratio (ICR), a Fisher-based metric that quantifies how residual variation contaminates invariant signal in feature space. We use this framework to analyze both discriminative and generative behavior of diffusion models. On the representation side, we find that invariance peaks at intermediate noise levels, which also yield the best downstream classification performance. On the generative side, we study how training transitions from genuine generalization to memorization in data-limited regimes, and show that ICR serves as a sensitive training-time indicator of early learning: increasing residual energy along Fisher directions marks the onset of memorization, detectable from training features alone without external evaluators or held-out test sets. Overall, our results show that diffusion models can be monitored from a self-supervised perspective through the geometry of their learned representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。