研究形状、纹理和颜色对语义分割学习的影响，发现形状+颜色组合提升小物体和边缘像素识别。
On the Influence of Shape, Texture and Color for Learning Semantic Segmentation
- 通过分离数据集中的视觉线索，训练各线索专家模型
- 形状与颜色结合在小物体和边界的分割上表现最佳
- 结果在卷积与变压器架构中一致，适合关注特征依赖的研究者
近期研究关注预训练深度神经网络在图像分类中的形状与纹理偏好。本文转向训练过程，分析在缺失其他线索时，深度神经网络从形状、纹理、颜色线索中分别能学到什么，及其联合影响对学习效果的作用。通过将数据集分解为特定线索版本，在多个层面进行分析。针对语义分割任务，从简化线索的数据集中学习，构建线索专家；通过早期融合构建合适数据集，结合晚期融合专家以研究线索在像素级的影响位置。在Cityscapes、PASCAL Context及合成的CARLA数据集上的实验表明，无单一线索占主导，但形状+颜色专家显著提升小物体与边界像素的预测性能。线索性能排序在测试的卷积与变换器架构中保持一致，显示二者具备相似的线索提取能力，尽管预训练变换器被认为比卷积网络更偏向形状。
原文摘要 · Abstract (English)
Recent research has investigated the shape and texture biases of pre-trained deep neural networks (DNNs) in image classification. Those works test how much a trained DNN relies on specific image cues like texture. The present study shifts the focus to understanding the cue influence during training, analyzing what DNNs can learn from shape, texture, and color cues in absence of the others; investigating their individual and combined influence on the learning success. We analyze these cue influences at multiple levels by decomposing datasets into cue-specific versions. Addressing semantic segmentation, we learn the given task from these reduced cue datasets, creating cue experts. Early fusion of cues is performed by constructing appropriate datasets. This is complemented by a late fusion of experts which allows us to study cue influence location-dependent on pixel level. Experiments on Cityscapes, PASCAL Context, and a synthetic CARLA dataset show that while no single cue dominates, the shape + color expert predominantly improves the prediction of small objects and border pixels. The cue performance order is consistent for the tested convolutional and transformer architecture, indicating similar cue extraction capabilities, although pre-trained transformers are said to be more biased towards shape than convolutional neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。