扩散模型难学图像特征间的隐藏规则,研究揭示其根本缺陷并提出改进方向。
Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
- 设计四类强相关特征合成任务,测试扩散模型对隐含规则的建模能力。
- 主流模型在光影、镜像等细粒度规则上表现不佳,存在系统性错误。
- 理论证明去噪得分匹配目标不兼容规则一致性,适合关注生成逻辑的研究者。
尽管扩散模型在数据生成中表现出色,但在某些场景下仍会出现输出不理想的问题。本文聚焦于其学习图像特征间隐藏关系的能力:对于具有依赖关系的特征(如太阳高度𝑥与影子长度𝑦),探究扩散模型能否准确捕捉条件分布𝑝(𝑦|𝑥)。对主流模型(如Stable Diffusion 3.5)的实证评估显示,存在光照-阴影关系不一致、物体-镜像反射错配等问题。为此,我们设计了四类具有强相关特征的合成任务,以评估模型的规则学习能力。实验表明,扩散模型可识别粗粒度规则,但难以处理细粒度规则。理论分析进一步证明,通过去噪得分匹配(DSM)训练的扩散模型在学习隐藏规则时存在恒定误差,因DSM目标与规则一致性不兼容。为缓解此问题,引入采样阶段的分类器引导,虽取得有限提升,但细粒度规则的微弱信号仍难以被分类器捕捉,为后续研究提供重要启示。
原文摘要 · Abstract (English)
Despite the remarkable success of diffusion models (DMs) in data generation, they exhibit specific failure cases with unsatisfactory outputs. We focus on one such limitation: the ability of DMs to learn hidden rules between image features. Specifically, for image data with dependent features ($\mathbf{x}$) and ($\mathbf{y}$) (e.g., the height of the sun ($\mathbf{x}$) and the length of the shadow ($\mathbf{y}$)), we investigate whether DMs can accurately capture the inter-feature rule ($p(\mathbf{y}|\mathbf{x})$). Empirical evaluations on mainstream DMs (e.g., Stable Diffusion 3.5) reveal consistent failures, such as inconsistent lighting-shadow relationships and mismatched object-mirror reflections. Inspired by these findings, we design four synthetic tasks with strongly correlated features to assess DMs' rule-learning abilities. Extensive experiments show that while DMs can identify coarse-grained rules, they struggle with fine-grained ones. Our theoretical analysis demonstrates that DMs trained via denoising score matching (DSM) exhibit constant errors in learning hidden rules, as the DSM objective is not compatible with rule conformity. To mitigate this, we introduce a common technique - incorporating additional classifier guidance during sampling, which achieves (limited) improvements. Our analysis reveals that the subtle signals of fine-grained rules are challenging for the classifier to capture, providing insights for future exploration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。