新基准测试发现大模型难从视觉示范中学会模糊空间推理。
Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?
- 设计新基准SVAT，用模糊空间任务考验模型上下文学习能力。
- 主流视觉语言模型零样本无法通过视觉示范学会新任务。
- 循序渐进加入简单数据可提升模型在上下文学习中的表现。
大型视觉语言模型（VLMs）在诸多计算机视觉任务中表现优异，常采用上下文学习（ICL）来适应新任务。但它们能否仅通过视觉示范学习新概念？还是仅限于模仿ICL示例的输出格式？本文提出新基准Spatial Visual Ambiguity Tasks（SVAT），挑战最先进的VLM在上下文学习中掌握新的视觉空间任务的能力。结果表明，多数VLM在零样本条件下失败，甚至微调后仍无法成功。然而，在训练中引入更简单的数据并采用课程学习策略，可显著提升其上下文学习性能。
原文摘要 · Abstract (English)
Large vision-language models (VLMs) have become state-of-the-art for many computer vision tasks, with in-context learning (ICL) as a popular adaptation strategy for new ones. But can VLMs learn novel concepts purely from visual demonstrations, or are they limited to adapting to the output format of ICL examples? We propose a new benchmark we call Spatial Visual Ambiguity Tasks (SVAT) that challenges state-of-the-art VLMs to learn new visuospatial tasks in-context. We find that VLMs fail to do this zero-shot, and sometimes continue to fail after finetuning. However, adding simpler data to the training by curriculum learning leads to improved ICL performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。