让模型自动找出图像中对分类最关键的局部区域,提升泛化能力。
PatchGen: Learning Soft Intra-Image Predictive Subsets for Visual Generalization

- 通过学习样本自适应的软掩码,定位图像中对分类有效的局部区域。
- 在自然图像与病理图像上均提升泛化性能,尤其在未知类别上表现更优。
- 无需文本监督,适用于缺乏标注数据的视觉任务场景。
视觉分类器需应对数据分布变化、目标类别变化及其组合,但现有方法多关注域不变性,忽视了图像内部预测充分性。本文提出假设:每张图像中存在一个样本自适应的最优局部预测子集,足以完成标签预测,而其余区域为非必要上下文,可能与标签相关。理论分析表明,仅使用该最优子集可保持全图表示的贝叶斯风险,并随子集大小减小而降低复杂度。基于此,我们提出PatchGen,一种无文本依赖模块,学习样本相关的软预测子集掩码,作为未观测最优掩码的任务驱动代理。组织学图像可视化显示,PatchGen对肿瘤一致区域赋予更高得分,而非频繁共现的炎症区域。在涵盖三类分布偏移的自然图像与病理图像基准测试中,PatchGen在多数配置下优于基线模型,提升对未知类别的泛化能力,且无需文本监督即媲美视觉-语言方法。
原文摘要 · Abstract (English)
Visual classifiers are expected to generalize under data shifts, target shifts, and their combinations, yet most existing methods focus on domain invariance while failing to address intra-image predictive sufficiency. We investigate the structural hypothesis that each image contains a sample-adaptive oracle intra-image predictive subset sufficient for label prediction, while the remaining patches form non-essential complementary context that may correlate with the label. The theoretical analysis shows that restricting prediction to this oracle subset preserves the Bayes risk achievable by the full-patch representation while admitting a complexity bound that tightens with the oracle-subset size. Based on this view, we propose PatchGen, a text-free module that learns a sample-dependent soft predictive-subset mask as a task-driven proxy for the unobserved oracle subset mask. Specifically, histopathology visualizations suggest that PatchGen assigns higher scores to tumor-consistent regions than to some frequently co-occurring inflammatory context. Extensive experiments on natural and histopathological image benchmarks spanning all three shift settings show that PatchGen improves average performance over matched-backbone baselines in most evaluated configurations, enhances generalization to unknown classes, and remains competitive with vision-language methods without text supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。