用扩散模型实现零样本手绘3D形状检索,提升抽象草图匹配能力。
Diff-SBSR: Learning Multimodal Feature-Enhanced Diffusion Models for Zero-Shot Sketch-Based 3D Shape Retrieval

- 利用冻结的Stable Diffusion和CLIP多模态特征增强草图表征
- 在两个公开数据集上达到领先性能,优于现有最先进方法
- 适合做零样本3D形状检索的研究者与开发者
本文首次探索将文本到图像扩散模型应用于零样本手绘3D形状检索(ZS-SBSR)。现有方法在无类别监督且草图输入极度稀疏的情况下表现不佳。我们的核心洞察是:大规模预训练扩散模型具备开放词汇能力和强形状偏好,适用于零样本视觉检索。我们采用冻结的Stable Diffusion骨干网络,从U-Net中间层提取并聚合草图与渲染3D视图的判别性表征。为克服扩散模型对高度抽象、稀疏草图的适应困难及与自然图像间的显著域差距,我们提出一种无需昂贵重训练的多模态特征增强策略:通过CLIP提供全局与局部视觉特征,并结合可学习软提示与BLIP生成的硬文本描述,增强语义上下文捕捉能力并聚焦草图轮廓。此外,采用Circle-T损失动态强化正样本吸引力,待负样本充分分离后,适应草图噪声,实现更有效的草图-3D对齐。在两个公开基准上的大量实验表明,本方法在ZS-SBSR任务中持续优于现有最先进方法。
原文摘要 · Abstract (English)
This paper presents the first exploration of text-to-image diffusion models for zero-shot sketch-based 3D shape retrieval (ZS-SBSR). Existing sketch-based 3D shape retrieval methods struggle in zero-shot settings due to the absence of category supervision and the extreme sparsity of sketch inputs. Our key insight is that large-scale pretrained diffusion models inherently exhibit open-vocabulary capability and strong shape bias, making them well suited for zero-shot visual retrieval. We leverage a frozen Stable Diffusion backbone to extract and aggregate discriminative representations from intermediate U-Net layers for both sketches and rendered 3D views. Diffusion models struggle with sketches due to their extreme abstraction and sparsity, compounded by a significant domain gap from natural images. To address this limitation without costly retraining, we introduce a multimodal feature-enhanced strategy that conditions the frozen diffusion backbone with complementary visual and textual cues from CLIP, explicitly enhancing the ability of semantic context capture and concentrating on sketch contours. Specifically, we inject global and local visual features derived from a pretrained CLIP visual encoder, and incorporate enriched textual guidance by combining learnable soft prompts with hard textual descriptions generated by BLIP. Furthermore, we employ the Circle-T loss to dynamically strengthen positive-pair attraction once negative samples are sufficiently separated, thereby adapting to sketch noise and enabling more effective sketch-3D alignment. Extensive experiments on two public benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches in ZS-SBSR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。