构建分层场景解析框架,让AI理解物体、部件与功能间的结构关系。
SceneParser: Hierarchical Scene Parsing for Visual Semantics Understanding

- 提出分层场景解析任务,显式建模场景-物体-部件-功能的层级绑定关系。
- 在包含174万条功能链的数据集上,模型实现更完整的结构化理解能力。
- 适合做机器人交互、智能导航等需要深层语义理解的场景应用。
通用场景感知已从物体识别发展到开放词汇定位、部件定位和功能预测。然而这些能力常表现为孤立的预测,无法捕捉交互所需的结构化依赖。为此,我们提出分层场景解析(Hierarchical Scene Parsing),以显式层级结构(场景→物体→部件→功能)表示物理场景,并建立跨层级绑定。我们构建了基于视觉语言模型的SceneParser,通过结构补全伪标签和课程学习实现统一分层生成。为支持训练与评估,我们构建了SceneParser-Bench,一个大规模基准数据集,包含11万张训练图像、5千张验证图像、77.7万物体、114万部件、174万功能标注及174万有效物体-部件-功能链实例。我们设计了从Level-1到Level-3的条件指标和ParseRate,用于评估定位、跨层级绑定与层级完整性。实验表明,现有多模态大模型与感知拼接流水线在该任务上表现不佳,而SceneParser展现出更强的结构感知性能。消融实验、在COCO与AGD20K上的评估以及下游规划探测进一步证明其兼容性与可操作性。
原文摘要 · Abstract (English)
General scene perception has progressed from object recognition toward open-vocabulary grounding, part localization, and affordance prediction. Yet these capabilities are often realized as isolated predictions that localize objects, parts, or interaction points without capturing the structured dependencies needed for interaction-oriented scene understanding. To address this gap, we introduce Hierarchical Scene Parsing, an interaction-oriented parsing task that represents physical scenes as explicit scene -> object -> part -> affordance hierarchies with cross-level bindings. We instantiate this task with SceneParser, a VLM-based parser trained for unified hierarchical generation with structural-completion pseudo labels and curriculum learning. To support training and evaluation, we construct SceneParser-Bench, a large-scale benchmark built with a scalable hierarchical data engine, containing 110K training images, a 5K validation split, 777K objects, 1.14M parts, 1.74M affordance annotations, and 1.74M valid object-part-affordance chain instances. We further introduce Level-1 to Level-3 conditional metrics and ParseRate to evaluate localization, cross-level binding, and hierarchical completeness. Experiments show that existing MLLMs and perception-stitching pipelines struggle with hierarchical parsing on our SceneParser-Bench, while SceneParser achieves stronger structure-aware performance. Besides, ablations, evaluations on COCO and AGD20K, and a downstream planning probe demonstrate that our SceneParser is compatible with conventional tasks and provides an actionable representation for visual understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。