测试拍摄参数对视觉模型性能的影响，发现数据集有严重偏差。
SNAP: A Benchmark for Testing the Effects of Capture Conditions on Fundamental Vision Tasks
- 构建SNAP基准，控制光照下密集采样相机参数
- 模型在正常曝光下仍远低于人类准确率
- 微小拍摄参数变化就显著影响模型表现
深度学习视觉算法在各类图像扰动下的泛化能力难以建立，仍是研究热点。以往分析多集中于已捕获图像，而图像形成流程和环境的影响研究较少。本文通过分析快门速度、感光度（ISO）和光圈等拍摄条件对图像分类、目标检测和视觉问答（VQA）任务的影响，评估常见视觉数据集的采集偏差。为此，我们创建了新基准SNAP（Shutter speed, ISO sensitivity, and APerture），包含在受控光照条件下、使用密集采样相机设置拍摄的物体图像。我们评估了大量深度学习视觉模型，揭示拍摄条件对各项任务的影响。最后，我们开展实验建立VQA任务的人类基线。结果表明，现有视觉数据集存在显著采集偏差，基于这些数据训练的模型即使在良好曝光图像上也未达到人类准确率，且对曝光变化及相机设置微小调整均敏感。代码与数据见https://github.com/ykotseruba/SNAP。
原文摘要 · Abstract (English)
Generalization of deep-learning-based (DL) computer vision algorithms to various image perturbations is hard to establish and remains an active area of research. The majority of past analyses focused on the images already captured, whereas effects of the image formation pipeline and environment are less studied. In this paper, we address this issue by analyzing the impact of capture conditions, such as camera parameters and lighting, on DL model performance on 3 vision tasks -- image classification, object detection, and visual question answering (VQA). To this end, we assess capture bias in common vision datasets and create a new benchmark, SNAP (for $\textbf{S}$hutter speed, ISO se$\textbf{N}$sitivity, and $\textbf{AP}$erture), consisting of images of objects taken under controlled lighting conditions and with densely sampled camera settings. We then evaluate a large number of DL vision models and show the effects of capture conditions on each selected vision task. Lastly, we conduct an experiment to establish a human baseline for the VQA task. Our results show that computer vision datasets are significantly biased, the models trained on this data do not reach human accuracy even on the well-exposed images, and are susceptible to both major exposure changes and minute variations of camera settings. Code and data can be found at https://github.com/ykotseruba/SNAP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。