arXiv:2501.17642cs.CV2025-01被引 5

提出高效方法减少开放词汇语义分割中的冗余,提升速度与精度。

Efficient Redundancy Reduction for Open-Vocabulary Semantic Segmentation

  • 构建紧凑类别词表与分层成本图,减少冗余信息。
  • 通过空间与类别级序列压缩,降低计算负担。
  • 在ADE20K-847上提速3.1倍,性能提升5.6%。

开放词汇语义分割(OVSS)是一项开放世界任务,旨在将图像中每个像素分配到由任意文本描述定义的类别。尽管大规模视觉-语言模型展现出强大的开放词汇能力,但其图像级预训练限制了在像素级密集预测任务上的表现。近期基于代价的方法通过构建像素-文本代价图并利用代价聚合机制缩小了粒度差距。然而,这些方法仍存在计算开销高、推理延迟长的问题。本文识别出代价框架中的两大冗余来源:代价图构建引入的冗余信息和代价聚合中低效的序列建模。为此,提出ERR-Seg,包含冗余减少的分层代价图(RRHC)与冗余减少的代价聚合(RRCA)。RRHC通过为每张图像定制紧凑类别词表并融合分层代价图增强语义表示;RRCA在聚合前进行空间与类别级序列压缩,显著降低计算量。整体上,ERR-Seg实现了轻量化设计,在不牺牲准确率的前提下大幅节省内存与计算资源。相比先前最先进方法,在ADE20K-847基准上性能提升5.6%,推理速度加快3.1倍。

原文摘要 · Abstract (English)

Open-vocabulary semantic segmentation (OVSS) is an open-world task that aims to assign each pixel within an image to a specific class defined by arbitrary text descriptions. While large-scale vision-language models have shown remarkable open-vocabulary capabilities, their image-level pretraining limits effectiveness on pixel-wise dense prediction tasks like OVSS. Recent cost-based methods narrow this granularity gap by constructing pixel-text cost maps and refining them via cost aggregation mechanisms. Despite achieving promising performance, these approaches suffer from high computational costs and long inference latency. In this paper, we identify two major sources of redundancy in the cost-based OVSS framework: redundant information introduced during cost maps construction and inefficient sequence modeling in cost aggregation. To address these issues, we propose ERR-Seg, an efficient architecture that incorporates Redundancy-Reduced Hierarchical Cost maps (RRHC) and Redundancy-Reduced Cost Aggregation (RRCA). Specifically, RRHC reduces redundant class channels by customizing a compact class vocabulary for each image and integrates hierarchical cost maps to enrich semantic representation. RRCA alleviates computational burden by performing both spatial-level and class-level sequence reduction before aggregation. Overall, ERR-Seg results in a lightweight structure for OVSS, characterized by substantial memory and computational savings without compromising accuracy. Compared to previous state-of-the-art methods on the ADE20K-847 benchmark, ERR-Seg improves performance by $5.6\%$ while achieving a 3.1$\times$ speedup.

语义分割视觉语言高效模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。