提出一种新方法,让扩散模型自动识别数据的内在维度。
On Convolutions, Intrinsic Dimension, and Diffusion Models
- 通过分析噪声添加过程中的密度变化,推导出内在维度估计器
- 证明该方法在真实数据分布下仍有效,突破了原有理想假设
- 适用于检测异常数据和生成内容,对安全与质量评估有帮助
高维数据(如图像)通常位于低维流形上。扩散模型通过逐步添加高斯噪声并学习逆过程来生成数据,被证实能捕捉低维支撑分布。对于流形上的任一数据点,理论上应可从中推断其局部内在维度(LID)。Kamkari等(2024b)提出FLIPD方法,通过噪声水平下对数边际密度的变化率估计LID,表现优于现有方法。然而该理论仅在仿射流形这一不现实假设下成立。本文填补该空白,证明了在更合理的假设下FLIPD依然正确;同时表明,将高斯卷积替换为均匀卷积后,类似结论仍成立,拓展了方法适用范围。
原文摘要 · Abstract (English)
The manifold hypothesis asserts that data of interest in high-dimensional ambient spaces, such as image data, lies on unknown low-dimensional submanifolds. Diffusion models (DMs) -- which operate by convolving data with progressively larger amounts of Gaussian noise and then learning to revert this process -- have risen to prominence as the most performant generative models, and are known to be able to learn distributions with low-dimensional support. For a given datum in one of these submanifolds, we should thus intuitively expect DMs to have implicitly learned its corresponding local intrinsic dimension (LID), i.e. the dimension of the submanifold it belongs to. Kamkari et al. (2024b) recently showed that this is indeed the case by linking this LID to the rate of change of the log marginal densities of the DM with respect to the amount of added noise, resulting in an LID estimator known as FLIPD. LID estimators such as FLIPD have a plethora of uses, among others they quantify the complexity of a given datum, and can be used to detect outliers, adversarial examples and AI-generated text. FLIPD achieves state-of-the-art performance at LID estimation, yet its theoretical underpinnings are incomplete since Kamkari et al. (2024b) only proved its correctness under the highly unrealistic assumption of affine submanifolds. In this work we bridge this gap by formally proving the correctness of FLIPD under realistic assumptions. Additionally, we show that an analogous result holds when Gaussian convolutions are replaced with uniform ones, and discuss the relevance of this result.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。