无需训练即可实现开放词汇语义分割,通过几何对齐提升图文一致性。
PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentation
- 先对齐后传播:用正交投影对齐文本与图像特征空间。
- 在标准基准上超越现有方法,无背景/有背景协议均领先。
- 仅需固定参数,推理轻量,可直接插入现有模型使用。
训练自由的开放词汇语义分割(OVSS)可在不重新训练的情况下快速适配新标签集。然而,许多方法依赖复杂的后处理,或孤立处理文本与视觉信息,忽视了跨模态几何关系。部分方法引入额外视觉主干或多模态流水线,增加复杂度和延迟,牺牲设计简洁性。本文提出PEARL——Procrustes对齐结合文本感知拉普拉斯传播,一种紧凑的两步推理流程,遵循对齐-传播原则。在最后自注意力块内,通过稳定的极坐标迭代执行正交投影,将键向量旋转至查询子空间。随后,文本感知拉普拉斯传播在小网格上通过置信加权、文本引导的图求解优化像素级逻辑值:文本提供数据可信信号与邻域门控,图像梯度保留边界。本方法完全训练自由、即插即用,仅使用固定常数,每头添加微小投影及少量共轭梯度步骤,带来极低延迟。在标准基准上,PEARL在无需额外数据或辅助主干的情况下,于有背景与无背景协议下均达到训练自由OVSS新最佳性能。
原文摘要 · Abstract (English)
Training-free open-vocabulary semantic segmentation (OVSS) promises rapid adaptation to new label sets without retraining. Yet, many methods rely on heavy post-processing or handle text and vision in isolation, leaving cross-modal geometry underutilized. Others introduce auxiliary vision backbones or multi-model pipelines, which increase complexity and latency while compromising design simplicity. We present PEARL, \textbf{\underline{P}}rocrust\textbf{\underline{e}}s \textbf{\underline{a}}lignment with text-awa\textbf{\underline{r}}e \textbf{\underline{L}}aplacian propagation, a compact two-step inference that follows an align-then-propagate principle. The Procrustes alignment step performs an orthogonal projection inside the last self-attention block, rotating keys toward the query subspace via a stable polar iteration. The text-aware Laplacian propagation then refines per-pixel logits on a small grid through a confidence-weighted, text-guided graph solve: text provides both a data-trust signal and neighbor gating, while image gradients preserve boundaries. In this work, our method is fully training-free, plug-and-play, and uses only fixed constants, adding minimal latency with a small per-head projection and a few conjugate-gradient steps. Our approach, PEARL, sets a new state-of-the-art in training-free OVSS without extra data or auxiliary backbones across standard benchmarks, achieving superior performance under both with-background and without-background protocols.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。