arXiv:2503.09399cs.CVcs.AI2025-03被引 2

通过拆解图像并重新组合,让模型摆脱对物体位置和背景的依赖。

ForAug: Mitigating Biases in Image Classification via Controlled Image Compositions

  • 将图像分解为前景与中性背景,自由控制物体位置、大小和背景类型。
  • 在ImageNet上提升最高6个百分点,在细粒度分类任务上提升7.3个百分点。
  • 可用来检测模型是否依赖背景或中心位置,适合研究模型鲁棒性的人使用。

大规模图像分类数据集存在强烈的组合偏差:物体常居中出现、具有特定尺度,并与类别相关的背景共现。模型利用这些偏差获得高分布内准确率,但在分布外表现脆弱。为此,我们提出ForAug,一种可控图像组成增强方法,将每张训练图像分解为前景对象和背景,再重新组合以显式操控物体位置、尺度和背景身份。ForAug使用现成的分割与修复模型(i)提取前景并生成中性背景,(ii)将前景贴到多样化中性背景上,再应用标准强增强策略。相比传统增强和内容混合方法,该分解方式提供了直接的控制变量,打破前景-背景相关性。在10种架构上,ForAug使ImageNet top-1准确率提升最高达6个百分点,在细粒度下游数据集上提升高达7.3个百分点。此外,相同的控制变量可用于定向诊断测试:通过受控背景替换与位置/尺度扫描,量化背景依赖、前景聚焦、中心偏差和尺寸偏差,结果显示ForAug显著减少这些捷径行为,并在标准分布偏移基准上将准确率提升最高达19个百分点。代码与数据集已公开于https://github.com/tobna/ForAug。

原文摘要 · Abstract (English)

Large-scale image classification datasets exhibit strong compositional biases: objects tend to be centered, appear at characteristic scales, and co-occur with class-specific context. By exploiting such biases, models attain high in-distribution accuracy but remain fragile under distribution shifts. To address this issue, we introduce ForAug, a controlled composition augmentation scheme that factorizes each training image into a foreground object and a background and recombines them to explicitly manipulate object position, object scale, and background identity. ForAug uses off-the-shelf segmentation and inpainting models to (i) extract the foreground and synthesize a neutral background, and (ii) paste the foreground onto diverse neutral backgrounds before applying standard strong augmentation policies. Compared to conventional augmentations and content-mixing methods, our factorization provides direct control knobs that break foreground-background correlations. Across 10 architectures, ForAug improves ImageNet top-1 accuracy by up to 6 percentage points (p.p.) and yields gains of up to 7.3 p.p. on fine-grained downstream datasets. Moreover, the same control knobs enable targeted diagnostic tests: we quantify background reliance, foreground focus, center bias, and size bias via controlled background swaps and position/scale sweeps, and show that training with ForAug substantially reduces these shortcut behaviors and significantly increases accuracy on standard distribution-shift benchmarks by up to $19$ p.p. Our code and dataset are publicly available at https://github.com/tobna/ForAug.

图像增强偏差缓解模型鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。