用低通滤波让视觉模型更像人脑,测试时模糊图像即可大幅提升表现。
Low-Pass Filtering Improves Behavioral Alignment of Vision Models
- 在测试时对图像做低通滤波,能显著提升模型与人类行为的一致性。
- 仅通过测试时模糊处理,就使模型对人类的对齐差距减半,达新最优。
- 最优滤波器的频率特性与人眼视觉系统匹配,解释了为何有效。
尽管深度神经网络在计算机视觉基准上表现优异,但在误差一致性与形状偏差等人类视觉行为建模方面仍存在不足。以往研究认为生成式分类器可大幅改善行为对齐,但本文发现,这种提升主要源于生成模型中一个看似无害的图像缩放操作,其本质是低通滤波。通过一系列受控实验,我们证明:在判别式模型(如CLIP)中去除高频空间信息,能显著提升行为对齐。仅在测试时对图像进行模糊处理——而非训练时使用模糊图像——即可在模型-人类对比基准上达到新状态,使当前模型与人类之间的对齐差距减半。此外,我们直接优化滤波器以最大化对齐性能,发现低通滤波器可能为最优。我们还首次计算出该基准下的帕累托最优解前沿。分析表明,最优高斯滤波器的频谱特征与人类视觉系统的带通滤波器近似匹配;人眼的对比敏感度函数(即检测正弦光栅所需最小对比度随空间/时间频率的变化)可用特定宽度的高斯滤波器良好拟合,且该宽度恰好对应最大误差一致性。
原文摘要 · Abstract (English)
Despite their impressive performance on computer vision benchmarks, Deep Neural Networks (DNNs) still fall short of adequately modeling human visual behavior, as measured by error consistency and shape bias. Recent work hypothesized that behavioral alignment can be drastically improved through \emph{generative} -- rather than \emph{discriminative} -- classifiers, with far-reaching implications for models of human vision. Here, we instead show that the increased alignment of generative models can be largely explained by a seemingly innocuous resizing operation in the generative model which effectively acts as a low-pass filter. In a series of controlled experiments, we show that removing high-frequency spatial information from discriminative models like CLIP drastically increases their behavioral alignment. Simply blurring images at test-time -- rather than training on blurred images -- achieves a new state-of-the-art score on the model-vs-human benchmark, halving the current alignment gap between DNNs and human observers. Furthermore, low-pass filters are likely optimal, which we demonstrate by directly optimizing filters for alignment. To contextualize the performance of optimal filters, we compute the frontier of all possible pareto-optimal solutions to the benchmark, which was formerly unknown. We explain our findings by observing that the frequency spectrum of optimal Gaussian filters roughly matches the spectrum of band-pass filters implemented by the human visual system. We show that the contrast sensitivity function, describing the inverse of the contrast threshold required for humans to detect a sinusoidal grating as a function of spatiotemporal frequency, is approximated well by Gaussian filters of the specific width that also maximizes error consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。