无需预设概念库,模型可自发现并组合新概念生成图像。
Test-Time Compositional Generalization in Diffusion Models via Concept Discovery

- 通过梯度上升从噪声得分中挖掘局部密度模式
- 在ColorMNIST和CelebA上超越基线方法20%以上性能
- 适合需要灵活生成新组合的开放场景应用
组合泛化要求模型能用熟悉元素生成新配置。现有扩散模型方法通常依赖预定义的概念或条件信号。本文提出,在测试时仅需一个分布外查询,即可利用模型对噪声边缘分布 $p_t(x_t)$ 学习的时间索引得分 $s_θ(x_t,t) \approx \nabla_{x_t}\log p_t(x_t)$,在多个去噪时间步进行梯度上升,恢复局部密度模态,将其映射到干净空间高斯分布,通过子模似然目标贪婪选择相关原型,并构建解析型乘积专家(PoE)教师模型。该教师模型可直接采样,或用于训练新类别嵌入与低秩适配器。在基于ColorMNIST和CelebA构建的保留组合基准上,两种方法均优于仅使用查询和最近训练类别的基线。结果表明,扩散模型的时间得分几何结构中蕴含可复用的密度模态概念,支持无需预设概念库的测试时组合生成。
原文摘要 · Abstract (English)
Compositional generalization requires models to produce novel configurations from familiar parts. In diffusion models, prior compositional generation methods typically assume that the relevant concepts or conditioning signals are already available. We instead ask whether a pretrained diffusion model can discover query-specific concepts from the time-indexed scores it learns for the noisy marginals $p_t(x_t)$ and compose them at test time. Given a single out-of-distribution query, our method performs gradient ascent on $s_θ(x_t,t) \approx \nabla_{x_t}\log p_t(x_t)$ at multiple noising timesteps to recover local density modes, maps these modes into clean-space Gaussians, greedily selects relevant prototypes with a submodular likelihood objective, and combines them into a product-of-experts (PoE) teacher model with an analytic score. This teacher model can be sampled directly through classifier-free guidance or used to generate a sample pool for training a new class embedding and low-rank adapter. On held-out composition benchmarks built from ColorMNIST and CelebA, both the analytic PoE sampler and the low-rank adapted model outperform query-only and nearest trained-class baselines. These results suggest that the time-indexed score geometry of the diffusion model contains reusable density-mode concepts that support test-time compositional generation without a predefined concept library.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。