用合成数据提升开放词汇全景分割性能，显著改善新类别泛化能力
DreamMask: Boosting Open-vocabulary Panoptic Segmentation with Synthetic Data
- 构建自动化合成数据流水线，支持词汇扩展与布局设计
- 融合真实与合成数据训练，使模型在ADE20K上提升2.1% mIoU
- 无需改造现有模型，可直接作为插件增强性能
开放词汇全景分割因实际应用价值受到广泛关注。然而，现有方法的性能提升主要依赖已知类别，对新类别的泛化能力不足。本文从数据视角出发，提出DreamMask，系统探索开放词汇场景下的数据生成与训练策略。首先，利用现成模型构建自动数据生成流水线，设计词汇扩展、布局排列、数据过滤等关键机制；生成数据显著优于人工收集的网络数据。其次，引入合成-真实对齐损失，弥合表征差距，显著提升多基准测试表现。整体上，DreamMask大幅简化大规模训练数据的获取，可作为即插即用模块增强现有方法。例如，在COCO上训练、ADE20K上测试时，模型性能超越此前最先进方法2.1% mIoU。
原文摘要 · Abstract (English)
Open-vocabulary panoptic segmentation has received significant attention due to its applicability in the real world. Despite claims of robust generalization, we find that the advancements of previous works are attributed mainly on trained categories, exposing a lack of generalization to novel classes. In this paper, we explore boosting existing models from a data-centric perspective. We propose DreamMask, which systematically explores how to generate training data in the open-vocabulary setting, and how to train the model with both real and synthetic data. For the first part, we propose an automatic data generation pipeline with off-the-shelf models. We propose crucial designs for vocabulary expansion, layout arrangement, data filtering, etc. Equipped with these techniques, our generated data could significantly outperform the manually collected web data. To train the model with generated data, a synthetic-real alignment loss is designed to bridge the representation gap, bringing noticeable improvements across multiple benchmarks. In general, DreamMask significantly simplifies the collection of large-scale training data, serving as a plug-and-play enhancement for existing methods. For instance, when trained on COCO and tested on ADE20K, the model equipped with DreamMask outperforms the previous state-of-the-art by a substantial margin of 2.1% mIoU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。