让视觉模型学会选真正有用的参考图，提升识别准确率。
Learning What Helps: Task-Aligned Context Selection for Vision Tasks
- 用联合优化方法训练选择器，让检索与任务目标对齐
- 在18个数据集上优于传统相似性匹配，尤其在小样本场景下
- 适合需要精准上下文参考的医学图像等高要求任务
人类常通过对比相关例子来解决视觉不确定性，但视觉变换器（ViTs）缺乏识别哪些例子能真正提升预测能力的能力。我们提出任务对齐上下文选择（TACS），一种学习选择真正有助于任务表现的配对例子而非仅外观相似例子的框架。TACS通过梯度监督与强化学习相结合的混合优化方案，联合训练选择器网络与任务模型，使检索成为学习目标的一部分。通过将选择与任务奖励对齐，TACS使判别模型能够发现哪些上下文例子确实有帮助。在涵盖细粒度识别、医学图像分类和医学图像分割的18个数据集上，TACS持续优于基于相似性的检索，尤其在挑战性或数据有限的设置中表现更优。
原文摘要 · Abstract (English)
Humans often resolve visual uncertainty by comparing an image with relevant examples, but ViTs lack the ability to identify which examples would improve their predictions. We present Task-Aligned Context Selection (TACS), a framework that learns to select paired examples which truly improve task performance rather than those that merely appear similar. TACS jointly trains a selector network with the task model through a hybrid optimization scheme combining gradient-based supervision and reinforcement learning, making retrieval part of the learning objective. By aligning selection with task rewards, TACS enables discriminative models to discover which contextual examples genuinely help. Across 18 datasets covering fine-grained recognition, medical image classification, and medical image segmentation, TACS consistently outperforms similarity-based retrieval, particularly in challenging or data-limited settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。