用2D框生成高质量视觉语言数据,提升模型空间推理能力
GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generation
- 仅用2D边界框生成数据,避免3D重建误差和幻觉
- 生成超850万对问答,人类验证准确率达91.16%
- 训练后模型在未见任务上泛化效果显著,提升超47%
视觉语言模型在多数任务中表现强劲,但在空间推理方面仍存短板。我们发现当前数据生成管道产出的数据人类验证率仅为57.6%,主要受限于单图3D重建的累积建模误差和宽泛答案容忍度,或基于描述的方法需高精度标注且易产生生成幻觉。本文提出GRAID,核心洞察是仅凭2D几何原语即可可靠判断空间关系。该方法仅依赖标准目标检测器输出的2D边界框,避免3D重建误差与生成幻觉,生成的数据经人工评估质量更高。我们在BDD100k、NuImages和Waymo数据集上应用该框架,生成超过850万条高质量视觉问答对,涵盖空间关系、计数、排序与大小比较等类型。其中一个数据集的人类验证准确率达91.16%,远高于同类工作57.6%。更重要的是,使用GRAID数据微调的模型能有效泛化:在6种问题类型上训练后,在超过10种未见类型上性能提升,对Llama 3.2B/11B模型在BDD和NuImages上的准确率分别提高47.5%和37.9%;在所有类型上训练后,还能在BLINK等现有基准上取得改进。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) achieve strong performance on many vision-language tasks but often struggle with spatial reasoning$\unicode{x2014}$a prerequisite for many applications. Empirically, we find that a dataset produced by a current training data generation pipeline has a 57.6% human validation rate. These rates stem from current limitations: single-image 3D reconstruction introduces cascading modeling errors and requires wide answer tolerances, while caption-based methods require hyper-detailed annotations and suffer from generative hallucinations. We present GRAID, built on the key insight that qualitative spatial relationships can be reliably determined from 2D geometric primitives alone. By operating exclusively on 2D bounding boxes from standard object detectors, GRAID avoids both 3D reconstruction errors and generative hallucinations, resulting in datasets that are of higher quality than existing tools that produce similar datasets as validated by human evaluations. We apply our framework to the BDD100k, NuImages, and Waymo datasets, generating over 8.5 million high-quality VQA pairs creating questions spanning spatial relations, counting, ranking, and size comparisons. We evaluate one of the datasets and find it achieves 91.16% human-validated accuracy$\unicode{x2014}$compared to 57.6% on a dataset generated by recent work. Critically, we demonstrate that when trained on GRAID data, models learn spatial reasoning concepts that generalize: models fine-tuned on 6 question types improve on over 10 held-out types, with accuracy gains of 47.5% on BDD and 37.9% on NuImages for Llama 3.2B 11B, and when trained on all questions types, achieve improvements on several existing benchmarks such as BLINK. The GRAID framework, datasets, and additional information can be found $\href{this https URL}{here}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。