arXiv:2506.07643cs.CV2025-06CVPR被引 11

用合成数据训练模型,让机器更懂图像中物体间的关系。

Synthetic Visual Genome

  • 用教师模型生成并筛选关系,构建高质量合成图像关系数据集
  • 30亿参数以下模型超越超3亿样本训练的同类模型,达88.9分领先水平
  • 适合研究视觉推理、场景图生成和多模态理解的开发者参考

视觉关系(空间、功能、交互、社交等)的推理被视为人类认知的基础。尽管多模态语言模型在视觉理解上取得进展,但对关系的精确推理与生成仍具挑战。我们提出ROBIN:一个通过密集标注关系指令微调的多模态语言模型,可大规模生成高质量密集场景图。为训练ROBIN,我们构建了SVG——一个合成场景图数据集,通过教师模型补全现有场景图中缺失的关系,并经精心设计的过滤流程确保质量。为实现任意图像的高精度、丰富场景图生成,我们引入SG-EDIT:一种自蒸馏框架,由GPT-4o进一步优化ROBIN预测结果,移除不合理关系或补充相关关系。整体数据集包含146,000张图像、560万条关系,覆盖260万个对象。实验显示,尽管仅用不到300万实例训练,ROBIN-3B模型在关系理解基准上超越训练量超3亿实例的同类模型,甚至优于最大至130亿参数的模型。尤其在指代表达理解任务中,达到88.9的最优得分,超越此前最高分87.4。结果表明,使用精细化场景图数据对保持多样视觉推理任务的高性能至关重要。

原文摘要 · Abstract (English)

Reasoning over visual relationships-spatial, functional, interactional, social, etc.-is considered to be a fundamental component of human cognition. Yet, despite the major advances in visual comprehension in multimodal language models (MLMs), precise reasoning over relationships and their generations remains a challenge. We introduce ROBIN: an MLM instruction-tuned with densely annotated relationships capable of constructing high-quality dense scene graphs at scale. To train ROBIN, we curate SVG, a synthetic scene graph dataset by completing the missing relations of selected objects in existing scene graphs using a teacher MLM and a carefully designed filtering process to ensure high-quality. To generate more accurate and rich scene graphs at scale for any image, we introduce SG-EDIT: a self-distillation framework where GPT-4o further refines ROBIN's predicted scene graphs by removing unlikely relations and/or suggesting relevant ones. In total, our dataset contains 146K images and 5.6M relationships for 2.6M objects. Results show that our ROBIN-3B model, despite being trained on less than 3 million instances, outperforms similar-size models trained on over 300 million instances on relationship understanding benchmarks, and even surpasses larger models up to 13B parameters. Notably, it achieves state-of-the-art performance in referring expression comprehension with a score of 88.9, surpassing the previous best of 87.4. Our results suggest that training on the refined scene graph data is crucial to maintaining high performance across diverse visual reasoning task.

视觉推理场景图生成多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。