arXiv:2511.16107cs.CVcs.AI2025-11

让视觉模型跨任务理解演示,用隐式文本引导完成图像编辑。

T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs

  • 通过隐式文本描述转换不匹配的演示,生成内容相关提示。
  • 在12个低层视觉任务上,跨任务性能优于固定提示方法。
  • 适合需要灵活适应新任务的图像编辑场景使用。

视觉上下文学习(VICL)通过少量输入输出示例完成视觉任务,无需模型训练。现有大型视觉语言模型(VLM)在示范与查询属于同一任务时表现良好,但在真实场景中,示范与查询常不匹配,导致模型难以判断是模仿已有变换还是从查询中推断新任务。本文研究跨任务VICL设定,提出T2T-VICL框架,通过协作式提示迁移,将不匹配的视觉示范转化为隐式文本指导,无需显式命名任务。大教师VLM首先生成视觉变化与任务差异的结构化描述,构建多样化隐式跨任务关系数据集;再将该能力蒸馏至轻量级学生VLM,使其从任务A的示范对和任务B的查询生成内容依赖提示。该提示用于引导冻结的图像编辑VLM,并引入基于评分的推理策略以排序候选结果。在12个低层视觉任务及超过20组跨任务对上的实验表明,T2T-VICL在任务感知对齐上持续优于固定提示,且常提升图像保真度,揭示了跨任务VICL的潜力与局限。代码已开源。

原文摘要 · Abstract (English)

Visual in-context learning (VICL) solves visual tasks by conditioning on a few input-output demonstrations without any model training. Recent advances in large vision-language models (VLMs) have shown promising VICL capability when the demonstration pair and the query belong to the same vision task, but real use cases often provide mismatched examples, making it unclear whether a VLM should imitate the demonstrated transformation or infer a new one from the query. This raises a fundamental question: Can VLMs perform cross-task VICL where demonstration and query differ? In the paper, we study this cross-task VICL setting and propose T2T-VICL, a collaborative prompt-transfer framework, which converts mismatched visual demonstrations into implicit textual guidance without explicitly naming the tasks. To do so, a large teacher VLM first generates structured descriptions of visual changes and task differences between task pairs, from which we construct a dataset of diverse implicit cross-task relations. We then distill this capability into a lightweight student VLM that produces content-dependent prompts from a task-A demonstration pair and a task-B query. The generated prompt is used to guide a frozen image-editing VLM, and a score-based inference strategy is introduced to rank multiple candidates. Experiments on 12 low-level vision tasks and over 20 evaluated cross-task pairs show that T2T-VICL consistently improves task-aware alignment over fixed prompting and often also improves image fidelity, revealing both the potential and limits of cross-task VICL. Our code is available on GitHub.

视觉上下文学习跨任务图像编辑提示蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。