无需重训练即可组合多个扩散模型,实现高效图像与分子生成。
The Superposition of Diffusion Models Using the Itô Density Estimator
- 通过伊藤密度估计器构建扩散模型叠加框架,理论基础扎实。
- 推理时仅需组合向量场,支持大规模模型且无额外计算开销。
- 适用于图像生成、编辑及蛋白质结构设计,结果更丰富可靠。
预训练扩散模型的爆发式增长催生了对无需重新训练即可组合多个模型方法的需求。本文提出一种名为超叠加(SuperDiff)的新框架,在生成阶段实现多模型组合。基于连续性方程从头推导,设计两种专用算法。SuperDiff引入一种可扩展的伊藤密度估计器来计算扩散SDE的对数似然,计算成本与哈钦森估计器相当,无需额外开销。该方法在推理时仅通过向量场复合完成叠加,具备良好可扩展性,且通过自动加权机制实现简便部署。实验表明,SuperDiff在推理时高效,并能模拟逻辑或与逻辑与操作。在CIFAR-10上生成更多样图像,在Stable Diffusion上实现更忠实的提示引导图像编辑,同时提升条件分子生成与无条件蛋白质新结构设计效果。
原文摘要 · Abstract (English)
The Cambrian explosion of easily accessible pre-trained diffusion models suggests a demand for methods that combine multiple different pre-trained diffusion models without incurring the significant computational burden of re-training a larger combined model. In this paper, we cast the problem of combining multiple pre-trained diffusion models at the generation stage under a novel proposed framework termed superposition. Theoretically, we derive superposition from rigorous first principles stemming from the celebrated continuity equation and design two novel algorithms tailor-made for combining diffusion models in SuperDiff. SuperDiff leverages a new scalable Itô density estimator for the log likelihood of the diffusion SDE which incurs no additional overhead compared to the well-known Hutchinson's estimator needed for divergence calculations. We demonstrate that SuperDiff is scalable to large pre-trained diffusion models as superposition is performed solely through composition during inference, and also enjoys painless implementation as it combines different pre-trained vector fields through an automated re-weighting scheme. Notably, we show that SuperDiff is efficient during inference time, and mimics traditional composition operators such as the logical OR and the logical AND. We empirically demonstrate the utility of using SuperDiff for generating more diverse images on CIFAR-10, more faithful prompt conditioned image editing using Stable Diffusion, as well as improved conditional molecule generation and unconditional de novo structure design of proteins. https://github.com/necludov/super-diffusion
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。