arXiv:2411.00425cs.CV2024-11被引 11

用AI生成真实场景缺陷,测试分割模型鲁棒性

Cityscape-Adverse: Benchmarking Robustness of Semantic Segmentation with Realistic Scene Modifications via Diffusion-Based Image Editing

  • 用扩散模型生成8类真实天气光照变化图像
  • CNN模型在极端条件下性能下降超30%,Transformer更稳定
  • 训练数据加入此基准可显著提升实际场景适应力

近期基于扩散模型的图像编辑技术,使得仅通过文本指令即可将图像转化为高度逼真的场景。本文提出Cityscape-Adverse基准,利用该技术模拟包括天气、光照和季节在内的八种不利条件,同时保持原始语义标签不变。我们评估了扩散模型生成逼真场景修改的可靠性,并测试了主流的卷积神经网络(CNN)与基于Transformer的语义分割模型在这些挑战性条件下的表现。结果表明,所有测试模型在极端条件下均出现显著性能下降,其中CNN架构受影响更大,而Transformer模型展现出更强韧性。我们还验证了在Cityscape-Adverse上训练的模型,在未见领域中应用时具备明显增强的鲁棒性。代码与数据集将发布于https://github.com/naufalso/cityscape-adverse。

原文摘要 · Abstract (English)

Recent advancements in generative AI, particularly diffusion-based image editing, have enabled the transformation of images into highly realistic scenes using only text instructions. This technology offers significant potential for generating diverse synthetic datasets to evaluate model robustness. In this paper, we introduce Cityscape-Adverse, a benchmark that employs diffusion-based image editing to simulate eight adverse conditions, including variations in weather, lighting, and seasons, while preserving the original semantic labels. We evaluate the reliability of diffusion-based models in generating realistic scene modifications and assess the performance of state-of-the-art CNN and Transformer-based semantic segmentation models under these challenging conditions. Additionally, we analyze which modifications have the greatest impact on model performance and explore how training on synthetic datasets can improve robustness in real-world adverse scenarios. Our results demonstrate that all tested models, particularly CNN-based architectures, experienced significant performance degradation under extreme conditions, while Transformer-based models exhibited greater resilience. We verify that models trained on Cityscape-Adverse show significantly enhanced resilience when applied to unseen domains. Code and datasets will be released at https://github.com/naufalso/cityscape-adverse.

语义分割扩散模型鲁棒性评测合成数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。