构建首个三层次场景理解数据集，支持动作、物体与关系联合分析。
SCLARO: A Dataset for Grounded Scenario-Level Scene Understanding and ScenarioCLIP for Benchmarking
- 设计分层编码器联合建模场景全局上下文、物体位置与物体间关系。
- 在61.5万张图像上实现零样本检索与跨域泛化性能领先。
- 适合做视觉推理、场景图生成与多模态理解的研究者使用。
在基于计算机视觉的精确现实场景理解范式中，对场景中物体、其相互关系及正在进行的动作进行联合推理是必要前提。然而，以往工作未同时涵盖三者，且缺乏大规模、多层次标注的数据集。为此，本文提出SCLARO（Scene-Contextual Localisation of Actions, Relations & Objects）数据集，包含615,805张覆盖室内、室外和驾驶场景的图像，标注有全局动作描述、物体边界框和关系三元组，提供结构化场景上下文。为基准测试该数据集，提出ScenarioCLIP模型，采用解耦编码器与EMA知识蒸馏，联合编码全局场景、物体和关系信息。在SCLARO上全面评估了零样本检索、线性探测、目标检测、谓词分类、场景图分类及跨域泛化等任务。相比PyramidCLIP等先前方法，ScenarioCLIP在物体与关系层面以及跨域泛化上表现更优。代码已开源。
原文摘要 · Abstract (English)
In the paradigm of computer vision-based precise real-world scene understanding, joint reasoning in terms of contextual understanding about the objects present in a scene, their inter-object relations, and the action being performed is an essential prerequisite. However, prior works have not addressed all three jointly, and no large-scale dataset provides grounded annotations at all three levels across diverse visual scenarios. Hence, this work introduces the SCLARO (Scene-Contextual Localisation of Actions, Relations & Objects) dataset, consisting of 615,805 images spanning indoor, outdoor, and driving scenarios, annotated with global action captions, object bounding boxes, and relation triplets that supply structured scene context beyond a free-text caption. To benchmark the dataset, we propose ScenarioCLIP, a tri-level reference model that jointly encodes global scene context, objects, and inter-object relations using disentangled encoders and EMA-based knowledge distillation. We benchmark across a comprehensive suite of tasks on the SCLARO Dataset, namely zero-shot retrieval, linear probe, object detection, predicate classification, scene-graph classification, and out-of-domain generalisation. ScenarioCLIP's disentangled encoders improve over the previous works, such as PyramidCLIP's shared encoder, most notably at the object and relation levels and on out-of-domain generalisation. Code for the data generation pipeline and ScenarioCLIP is available at https://github.com/scenario-clip/SCLARO-ScenarioCLIP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。