用合成图像提升视觉模型的空间感知能力
VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images
- 通过任务关键词自动生成合成图像与问答对
- 在10个视觉感知任务上提升7%-10%准确率
- 无需真实图像或人工标注,适合模型训练优化
视觉语言模型(VLMs)在空间理解与视角识别等视觉感知任务中仍表现不佳。一个可能原因是自然图像数据集对低层视觉技能的监督有限。为此,我们提出VisionFoundry,一种仅需任务名称即可生成合成数据的管道:利用大语言模型(LLMs)生成问题、答案和文生图(T2I)提示,再通过T2I模型合成图像,并用专有VLM验证一致性,全程无需参考图像或人工标注。基于此,我们构建了VisionFoundry-10K,一个包含10,000个图像-问题-答案三元组的合成视觉问答数据集,覆盖10个任务。在该数据集上训练的模型,在MMVP和CV-Bench-3D基准上分别取得+7%和+10%的性能提升,同时保持广泛能力,并展现出随数据量增长的良好可扩展性。结果表明,任务定向监督不足是关键瓶颈,而合成监督是系统化训练VLM的可行路径。
原文摘要 · Abstract (English)
Vision-language models (VLMs) still struggle with visual perception tasks such as spatial understanding and viewpoint recognition. One plausible contributing factor is that natural image datasets provide limited supervision for low-level visual skills. This motivates a practical question: can targeted synthetic supervision, generated from only a task keyword such as Depth Order, address these weaknesses? To investigate this question, we introduce VisionFoundry, a task-aware synthetic data generation pipeline that takes only the task name as input and uses large language models (LLMs) to generate questions, answers, and text-to-image (T2I) prompts, then synthesizes images with T2I models and verifies consistency with a proprietary VLM, requiring no reference images or human annotation. Using VisionFoundry, we construct VisionFoundry-10K, a synthetic visual question answering (VQA) dataset containing 10k image-question-answer triples spanning 10 tasks. Models trained on VisionFoundry-10K achieve substantial improvements on visual perception benchmarks: +7% on MMVP and +10% on CV-Bench-3D, while preserving broader capabilities and showing favorable scaling behavior as data size increases. Our results suggest that limited task-targeted supervision is an important contributor to this bottleneck and that synthetic supervision is a promising path toward more systematic training for VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。