融合CLIP与Stable Diffusion，实现通用草图特征提取
SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models
- 用CLIP语义特征动态注入SD去噪过程，补偿其频域偏差
- 在草图检索、识别、分割等任务上提升3.35%至29.42%
- 无需重训练，适合希望快速部署草图理解模型的研究者
尽管基础模型已革新计算机视觉，但其在草图理解上的表现受限于抽象稀疏的视觉输入。系统分析发现：Stable Diffusion（SD）难以从抽象草图中提取有效特征（相比照片效果差），且存在显著的频域偏差，抑制了草图理解所需的关键低频成分。我们不采用昂贵的重训练，而是通过策略性融合SD与具备强语义理解能力的CLIP，以弥补SD的空间-频率偏差。通过在SD去噪过程中动态注入CLIP特征，并自适应聚合跨语义层级特征，本方法在草图检索（+3.35%）、识别（+1.06%）、分割（+29.42%）和对应关系学习（+21.22%）上达到当前最优性能，首次实现了基础模型时代真正的通用草图特征表示。
原文摘要 · Abstract (English)
While foundation models have revolutionised computer vision, their effectiveness for sketch understanding remains limited by the unique challenges of abstract, sparse visual inputs. Through systematic analysis, we uncover two fundamental limitations: Stable Diffusion (SD) struggles to extract meaningful features from abstract sketches (unlike its success with photos), and exhibits a pronounced frequency-domain bias that suppresses essential low-frequency components needed for sketch understanding. Rather than costly retraining, we address these limitations by strategically combining SD with CLIP, whose strong semantic understanding naturally compensates for SD's spatial-frequency biases. By dynamically injecting CLIP features into SD's denoising process and adaptively aggregating features across semantic levels, our method achieves state-of-the-art performance in sketch retrieval (+3.35%), recognition (+1.06%), segmentation (+29.42%), and correspondence learning (+21.22%), demonstrating the first truly universal sketch feature representation in the era of foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。