用CLIP的语义能力提升对称检测，效果超越现有方法。
CLIPSym: Delving into Symmetry Detection with CLIP
- 结合图像语言模型与旋转等变解码器，联合学习对称性特征。
- 在DENDI、SDRW、LDRS三个数据集上均达到新最优性能。
- 提出语义感知提示分组，更好利用自然描述中的对称线索。
对称性是计算机视觉中最基本的几何线索之一，其检测一直是个挑战。随着视觉语言模型（如CLIP）的发展，我们探究预训练的CLIP能否通过利用自然图像描述中的对称性线索来辅助对称检测。为此提出CLIPSym，融合CLIP的图像与语言编码器，并采用基于Transformer与$G$-Convolution混合的旋转等变解码器，以检测旋转和反射对称性。为充分挖掘CLIP语言编码器潜力，我们设计了新型提示技术——语义感知提示分组（SAPG），通过聚合高频物体相关提示，增强语义线索对称检测的整合。实验表明，CLIPSym在三个标准对称检测数据集（DENDI、SDRW、LDRS）上均优于当前最先进方法。详细消融实验验证了预训练、等变解码器及SAPG技术的有效性。代码已开源：https://github.com/timyoung2333/CLIPSym。
原文摘要 · Abstract (English)
Symmetry is one of the most fundamental geometric cues in computer vision, and detecting it has been an ongoing challenge. With the recent advances in vision-language models,~i.e., CLIP, we investigate whether a pre-trained CLIP model can aid symmetry detection by leveraging the additional symmetry cues found in the natural image descriptions. We propose CLIPSym, which leverages CLIP's image and language encoders and a rotation-equivariant decoder based on a hybrid of Transformer and $G$-Convolution to detect rotation and reflection symmetries. To fully utilize CLIP's language encoder, we have developed a novel prompting technique called Semantic-Aware Prompt Grouping (SAPG), which aggregates a diverse set of frequent object-based prompts to better integrate the semantic cues for symmetry detection. Empirically, we show that CLIPSym outperforms the current state-of-the-art on three standard symmetry detection datasets (DENDI, SDRW, and LDRS). Finally, we conduct detailed ablations verifying the benefits of CLIP's pre-training, the proposed equivariant decoder, and the SAPG technique. The code is available at https://github.com/timyoung2333/CLIPSym.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。