测试大模型对3D形状的理解能力，发现其有抽象认知但远不如人类。
Do large language vision models understand 3D shapes?
- 用大量CGI图像测试模型识别相同3D形状的能力。
- 模型在方向或材质变化时表现尚可，但两者同时变时准确率大幅下降。
- 适合关注视觉理解局限性的研究人员和开发者参考。
大型视觉语言模型（LVLM）是实现世界通用视觉理解的主流AI方法，如GPT、Claude、Gemini和LLama等可通过图像分析复杂视觉场景。3D物体和形状是世界的基石，识别它们是人类感知的基础。本研究旨在测试LVLM是否真正理解3D形状，通过检验模型识别同一3D形状但不同朝向和材质/纹理的能力。使用大量基于CGI生成的图像，涵盖极多样化的物体、材质与场景。测试结果表明，模型匹配3D形状的能力显著低于人类，但远高于随机猜测，说明模型已获得一定抽象理解，但仍与人类差距巨大。主要发现：模型能较好识别同物体不同朝向的情况，以及同朝向但材质不同的相同3D形状；然而当物体材质与朝向均改变时，所有模型表现均远逊于人类。代码与基准数据集已公开。
原文摘要 · Abstract (English)
Large vision language models (LVLM) are the leading A.I approach for achieving a general visual understanding of the world. Models such as GPT, Claude, Gemini, and LLama can use images to understand and analyze complex visual scenes. 3D objects and shapes are the basic building blocks of the world, recognizing them is a fundamental part of human perception. The goal of this work is to test whether LVLMs truly understand 3D shapes by testing the models ability to identify and match objects of the exact same 3D shapes but with different orientations and materials/textures. A large number of test images were created using CGI with a huge number of highly diverse objects, materials, and scenes. The results of this test show that the ability of such models to match 3D shapes is significantly below humans but much higher than random guesses. Suggesting that the models have gained some abstract understanding of 3D shapes but still trail far beyond humans in this task. Mainly it seems that the models can easily identify the same object with a different orientation as well as matching identical 3D shapes of the same orientation but with different materials and textures. However, when both the object material and orientation are changed, all models perform poorly relative to humans. Code and benchmark are available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。