用连续轨迹对齐视觉与文本特征,提升开放词汇语义分割性能
DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation

- 通过微分方程建模文本嵌入向视觉流形的连续演化
- 在多个基准上达到当前最佳效果,显著优于现有方法
- 适合需要高精度跨模态对齐的开放词汇分割任务
开放词汇语义分割(OVSS)利用文本语义对预定义类别之外的物体进行分割。尽管自监督模型DINOv3提供了强大的结构化视觉表征,但其缺乏原生文本对齐,限制了其在OVSS中的直接应用。为此,我们提出DINOde,一种基于常微分方程(ODE)的框架,持续将CLIP文本嵌入与DINO视觉流形对齐。该方法包含两个互补组件:(i) 语义文本流(STF),通过连续ODE轨迹将文本嵌入演化至DINO流形;(ii) 全局上下文流(GCF),逐步优化由DINO CLS token承载的图像整体表征。为保持特征空间的超球面几何结构,我们引入速度切向投影,约束学习到的速度场位于切空间内。通过将对齐建模为连续轨迹,DINOde避免了离散MLP投影固有的流形缠结问题,实现了更鲁棒的跨模态对齐。大量实验表明,DINOde在多个OVSS基准上持续优于现有方法,达到当前最优性能。代码已开源:https://github.com/yoon307/DINOde。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation (OVSS) leverages textual semantics to segment objects beyond predefined categories. While the self-supervised model DINOv3 provides strong structured visual representations, its lack of native textual alignment hinders its direct application to OVSS. To bridge this gap, we propose DINOde, an ODE-based framework that continuously aligns CLIP text embeddings with the DINO visual manifold. Our approach employs two complementary components: (i) Semantic Text Flow (STF), which evolves text embeddings toward the DINO manifold through a continuous ODE trajectory, and (ii) Global Context Flow (GCF), which progressively refines the holistic image representation carried by DINO's CLS token. To preserve the hyperspherical geometry of the feature space during this evolution, we further introduce Velocity Tangent Projection, which constrains the learned velocity field to the tangent space. By modeling alignment as a continuous trajectory, DINOde avoids the manifold entanglement inherent in discrete MLP projections and yields more robust cross-modal alignment. Extensive experiments demonstrate that DINOde consistently outperforms existing methods and achieves state-of-the-art performance across multiple OVSS benchmarks. The code is available at https://github.com/yoon307/DINOde.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。