用最优传输理论提升开放词汇语义分割的视觉-语言对齐效果
OV-COAST: Cost Aggregation with Optimal Transport for Open-Vocabulary Semantic Segmentation
- 基于最优传输构建代价体，量化视觉与语言特征分布距离
- 两阶段优化使CAT-Seg在MESS上提升1.72% mIoU，SAN-B提升4.9%
- 适合关注开放词汇分割泛化能力的研究者
开放词汇语义分割（OVSS）旨在利用文本描述为图像中每个像素分配语义标签，通常依赖如CLIP等世界模型。为增强跨域泛化能力，本文提出基于最优传输的代价聚合方法（OV-COAST）。通过代价体构建成本矩阵，以最优传输理论对齐视觉-语言特征分布。采用两阶段优化：第一阶段用Sinkhorn距离求解最优传输问题，获得对齐解；第二阶段用该解指导CAT-Seg模型训练。在MESS基准测试中，所提方法显著提升使用ViT-B骨干网络的CAT-Seg性能，mIoU较原模型提升1.72%，较SAN-B提升4.9%。代码已开源。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation (OVSS) entails assigning semantic labels to each pixel in an image using textual descriptions, typically leveraging world models such as CLIP. To enhance out-of-domain generalization, we propose Cost Aggregation with Optimal Transport (OV-COAST) for open-vocabulary semantic segmentation. To align visual-language features within the framework of optimal transport theory, we employ cost volume to construct a cost matrix, which quantifies the distance between two distributions. Our approach adopts a two-stage optimization strategy: in the first stage, the optimal transport problem is solved using cost volume via Sinkhorn distance to obtain an alignment solution; in the second stage, this solution is used to guide the training of the CAT-Seg model. We evaluate state-of-the-art OVSS models on the MESS benchmark, where our approach notably improves the performance of the cost-aggregation model CAT-Seg with ViT-B backbone, achieving superior results, surpassing CAT-Seg by 1.72 % and SAN-B by 4.9 % mIoU. The code is available at https://github.com/adityagandhamal/OV-COAST/}{https://github.com/adityagandhamal/OV-COAST/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。