用零样本学习实现透明玻璃的精准分割,提升复杂场景下的识别效果。
Semantic Segmentation of Transparent and Opaque Drinking Glasses with the Help of Zero-shot Learning
- 基于Trans4Trans改进零样本模型,专攻透明物体分割
- 合成数据集+真实数据验证,使平均交并比提升13.68%
- 融合SAM2结果,对相似或遮挡玻璃赋予多类别标签
透明物体在图像中难以与背景区分,如各种形状和尺寸的饮酒杯。本文提出TransCaGNet,基于零样本模型CaGNet改进,将分割主干替换为Trans4Trans以增强透明物体识别能力。针对罕见玻璃类别的训练缺失问题,采用零样本学习实现未见类别的语义分割。构建了一个覆盖多样环境条件的新型合成数据集,并采集真实世界评估数据集。相比Zeg-Clip,TransCaGNet在多数类别上取得更高均值IoU和准确率;而在未见类别上Zeg-Clip表现更优。为进一步提升性能,结合模型语义分割与SAM 2结果,通过给玻璃分配多个可能类别来缓解类间相似、视角或遮挡带来的挑战。该方法使合成数据集上均值IoU提升最高达13.68%,均值准确率提升最高达17.88%。使用该困难合成数据集训练后,真实世界数据集上均值IoU提升最高5.55%,均值准确率提升最高5.72%。
原文摘要 · Abstract (English)
Segmenting transparent structures in images is challenging since they are difficult to distinguish from the background. Common examples are drinking glasses, which are a ubiquitous part of our lives and appear in many different shapes and sizes. In this work we propose TransCaGNet, a modified version of the zero-shot model CaGNet. We exchange the segmentation backbone with the architecture of Trans4Trans to be capable of segmenting transparent objects. Since some glasses are rarely captured, we use zeroshot learning to be able to create semantic segmentations of glass categories not given during training. We propose a novel synthetic dataset covering a diverse set of different environmental conditions. Additionally we capture a real-world evaluation dataset since most applications take place in the real world. Comparing our model with Zeg-Clip we are able to show that TransCaGNet produces better mean IoU and accuracy values while ZegClip outperforms it mostly for unseen classes. To improve the segmentation results, we combine the semantic segmentation of the models with the segmentation results of SAM 2. Our evaluation emphasizes that distinguishing between different classes is challenging for the models due to similarity, points of view, or coverings. Taking this behavior into account, we assign glasses multiple possible categories. The modification leads to an improvement up to 13.68% for the mean IoU and up to 17.88% for the mean accuracy values on the synthetic dataset. Using our difficult synthetic dataset for training, the models produce even better results on the real-world dataset. The mean IoU is improved up to 5.55% and the mean accuracy up to 5.72% on the real-world dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。