测试大模型理解与生成场景图的能力,发现生成复杂场景图时表现不佳。
LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Study
- 构建TSG Bench基准,评估大模型理解与生成场景图能力。
- 11个大模型在生成场景图任务中表现较差,尤其面对复杂叙事。
- 关键瓶颈是无法有效拆分复杂叙述中的离散场景,适合做视觉推理研究者参考。
大型语言模型(LLMs)卓越的推理与泛化能力推动其在具身智能、机器人等真实世界任务中的应用。为支持这些应用,对多模态环境中空间与时间的理解至关重要。近期工作利用场景图(scene graph)这一结构化表示来编码场景中的实体、属性及其关系。然而,对LLMs使用场景图能力的系统评估仍显不足。本文提出Text-Scene Graph (TSG) Bench基准,系统评估LLMs在(1)理解场景图和(2)从文本描述生成场景图两方面的表现。基于该基准,我们评估了11个LLMs,发现尽管模型在场景图理解上表现良好,但在复杂叙事生成场景图时存在显著困难。分析表明,模型未能有效从复杂叙述中分解出离散场景,成为生成瓶颈。研究结果凸显了改进场景图生成方法的必要性,并为未来研究提供了重要洞见。演示地址:https://tsg-bench.netlify.app;代码与数据公开于:https://github.com/docworlds/tsg-bench。
原文摘要 · Abstract (English)
The remarkable reasoning and generalization capabilities of Large Language Models (LLMs) have paved the way for their expanding applications in embodied AI, robotics, and other real-world tasks. To effectively support these applications, grounding in spatial and temporal understanding in multimodal environments is essential. To this end, recent works have leveraged scene graphs, a structured representation that encodes entities, attributes, and their relationships in a scene. However, a comprehensive evaluation of LLMs' ability to utilize scene graphs remains limited. In this work, we introduce Text-Scene Graph (TSG) Bench, a benchmark designed to systematically assess LLMs' ability to (1) understand scene graphs and (2) generate them from textual narratives. With TSG Bench we evaluate 11 LLMs and reveal that, while models perform well on scene graph understanding, they struggle with scene graph generation, particularly for complex narratives. Our analysis indicates that these models fail to effectively decompose discrete scenes from a complex narrative, leading to a bottleneck when generating scene graphs. These findings underscore the need for improved methodologies in scene graph generation and provide valuable insights for future research. The demonstration of our benchmark is available at https://tsg-bench.netlify.app. Additionally, our code and evaluation data are publicly available at https://github.com/docworlds/tsg-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。