用可变长编码将分割掩码转为离散序列，实现图像视频统一分割。
Tokenizing Semantic Segmentation with Run Length Encoding
- 用运行长度编码将分割图转为离散令牌序列，适配语言模型生成。
- 在有限算力下仍达到顶尖模型水平，视频分割性能显著提升。
- 支持实例信息融合，适用于全景分割，代码开源便于复现。
本文提出一种统一的图像与视频语义分割方法，通过语言建模将分割掩码输出为离散令牌序列。采用运行长度编码（RLE）对分割掩码进行离散化，并基于Pix2Seq框架训练自回归模型生成这些令牌。设计新型令牌化策略压缩序列长度，使该方法可扩展至视频场景。同时，将实例信息融入令牌化过程，实现全景分割。在两个特定领域数据集上评估，尽管受限于计算资源，模型在某些场景下仍具竞争力。文章还提出若干改进方向以提升通用应用潜力，并公开代码与模型供研究使用。
原文摘要 · Abstract (English)
This paper presents a new unified approach to semantic segmentation in both images and videos by using language modeling to output the masks as sequences of discrete tokens. We use run length encoding (RLE) to discretize the segmentation masks, and adapt the Pix2Seq framework to learn autoregressive models to output these tokens. We propose novel tokenization strategies to compress the lengths of the token sequences to make it practicable to extend this approach to videos. We also show how instance information can be incorporated into the tokenization process to perform panoptic segmentation. We evaluate our models on two domain-specific datasets to demonstrate their competitiveness with the state of the art in certain scenarios, in spite of being severely bottlenecked by our limited computational resources. We supplement these analyses by proposing several promising approaches to foster future competitiveness in general-purpose applications, and facilitate this by making our code and models publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。