用几何结构优化图像语义对应,解决对称与重复带来的模糊问题。
Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild
- 将伪标签生成建模为融合型格罗莫夫-沃瑟斯坦问题,兼顾特征相似与结构一致性。
- 在SPair-71k和AP-10k上达到新最优,无需显式几何标注。
- 适合做无标注图像语义对应任务的研究者与开发者。
语义对应对于处理缺乏显式对应标注的多样化真实场景图像至关重要。尽管近期2D基础模型提供强大特征,但通过最近邻伪标签进行无监督学习存在局限:仅局部操作,忽略结构关系,且依赖2D外观无法解决由对称性或重复特征引发的几何歧义。本文提出Shape-of-You(SoY),将伪标签生成重构为融合型格罗莫夫-沃瑟斯坦(FGW)问题,联合优化特征间相似性与内部结构一致性。该框架利用3D基础模型在几何空间中定义结构,化解上述歧义。由于FGW是计算代价高昂的二次问题,我们通过锚点线性化近似求解,得到一个结构一致但含噪的运输计划。为此设计软目标损失,动态融合该计划与网络预测,构建对噪声鲁棒的学习框架。SoY在SPair-71k和AP-10k数据集上取得当前最佳性能,建立了无需显式几何标注的语义对应新基准。代码已公开。
原文摘要 · Abstract (English)
Semantic correspondence is essential for handling diverse in-the-wild images lacking explicit correspondence annotations. While recent 2D foundation models offer powerful features, adapting them for unsupervised learning via nearest-neighbor pseudo-labels has key limitations: it operates locally, ignoring structural relationships, and consequently its reliance on 2D appearance fails to resolve geometric ambiguities arising from symmetries or repetitive features. In this work, we address this by reformulating pseudo-label generation as a Fused Gromov-Wasserstein (FGW) problem, which jointly optimizes inter-feature similarity and intra-structural consistency. Our framework, Shape-of-You (SoY), leverages a 3D foundation model to define this intra-structure in the geometric space, resolving abovementioned ambiguity. However, since FGW is a computationally prohibitive quadratic problem, we approximate it through anchor-based linearization. The resulting probabilistic transport plan provides a structurally consistent but noisy supervisory signal. Thus, we introduce a soft-target loss dynamically blending guidance from this plan with network predictions to build a learning framework robust to this noise. SoY achieves state-of-the-art performance on SPair-71k and AP-10k datasets, establishing a new benchmark in semantic correspondence without explicit geometric annotations. Code is available at Shape-of-You.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。