将图像提示嵌入看作点云，用最优传输实现更平滑的图像插值。
Which Way from B to A: The role of embedding geometry in image interpolation for Stable Diffusion
- 把CLIP嵌入视为点云，转为最优传输问题求最短路径
- 插值图像更平滑连贯，视觉质量显著提升
- 适合研究生成模型几何结构或图像过渡优化的人
研究表明，Stable Diffusion 对对比语言-图像预训练（CLIP）嵌入矩阵的行具有排列不变性。这启发我们将其自然地视为沃尔沙斯特（Wasserstein）空间中的点云，而非欧几里得空间中的矩阵。这一视角为理解嵌入空间几何提供了新可能。例如，在两个不同提示的嵌入之间进行插值时，我们将问题重新建模为最优传输问题。通过求解该问题，计算出连接嵌入的最短路径（测地线），从而在嵌入空间中实现更自然、几何上平滑的过渡。使用Stable Diffusion生成模型渲染时，这种路径产生的中间图像更加平滑且连贯。实验对比了基于最优传输的方法与其他标准插值方法生成图像的质量，结果表明，新的最优传输方法确实实现了更优的图像插值效果，表明将嵌入视为点云（而非矩阵）能更准确地反映并利用嵌入空间的几何结构。
原文摘要 · Abstract (English)
It can be shown that Stable Diffusion has a permutation-invariance property with respect to the rows of Contrastive Language-Image Pretraining (CLIP) embedding matrices. This inspired the novel observation that these embeddings can naturally be interpreted as point clouds in a Wasserstein space rather than as matrices in a Euclidean space. This perspective opens up new possibilities for understanding the geometry of embedding space. For example, when interpolating between embeddings of two distinct prompts, we propose reframing the interpolation problem as an optimal transport problem. By solving this optimal transport problem, we compute a shortest path (or geodesic) between embeddings that captures a more natural and geometrically smooth transition through the embedding space. This results in smoother and more coherent intermediate (interpolated) images when rendered by the Stable Diffusion generative model. We conduct experiments to investigate this effect, comparing the quality of interpolated images produced using optimal transport to those generated by other standard interpolation methods. The novel optimal transport--based approach presented indeed gives smoother image interpolations, suggesting that viewing the embeddings as point clouds (rather than as matrices) better reflects and leverages the geometry of the embedding space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。