测试物体中心表征在可控编辑下的语义忠实性,揭示现有模型的隐藏缺陷。
EditCLEVR: A Paired-Scene Intervention Benchmark for Compositional Faithfulness of Object-Centric Representations
- 构建配对场景编辑基准,精准控制单物体属性变化
- 提出SGIA和Delta-SGIA指标,区分代码空间移动与属性正确性
- 发现真实掩码下仍存在语义退化,局部稳定性不等于语义忠实
物体中心学习旨在将场景表示为可复用组合的物体。现有评估多关注分割、单图因子预测或下游准确率,但未直接检验对象表征在受控语义编辑下的行为是否正确。本文提出EditCLEVR,一个成对场景干预基准,每个样本包含相同物体索引与场景布局的前后两幅CLEVR风格图像,仅改变一个已知物体的单一属性,或无编辑重渲染用于漂移测量。该协议包含无需探针的表征变化定位与稳定性诊断,以及基于探针解码的语义忠实度指标,测试预测场景变化是否匹配预期干预,覆盖分布内与组合外分布(OOD)场景,可分别评估代码空间移动与解码后物体-属性正确性。引入场景图干预准确率(SGIA),要求完整后场景预测正确且唯一语义变化为意图编辑;另设Delta-SGIA作为辅助诊断,仅检查单点变化模式而不要求完整场景图正确。在真实掩码骨干、学习槽模型、SAM 2 + 冻结ViT模型及一种掩码-特征混合模型上的基线评估表明:即使使用真实掩码,CoGenT-OOD-core性能仍会下降;掩码来源仅部分解释原始性能差异;局部性或稳定性单独表现会夸大语义忠实度。代码已开源。
原文摘要 · Abstract (English)
Object-centric learning aims to represent scenes as objects whose properties can be reused in new combinations. Existing evaluations usually score segmentation, single-image factor prediction, or downstream accuracy, but these tests do not directly ask whether a per-object representation behaves correctly under a controlled semantic edit. We introduce EditCLEVR, a paired-scene intervention benchmark in which each example contains a before/after pair of CLEVR-style renders with the same object indices and scene layout, and either exactly one known attribute change on one known object or a no-edit re-render for drift measurement. The protocol includes probe-free diagnostics for representation-change localization and stability, together with probe-decoded semantic faithfulness metrics that test whether the predicted scene change matches the intended intervention across in-distribution and compositional out-of-distribution (OOD) suites, allowing code-space movement and decoded object-attribute correctness to be evaluated separately. We introduce the semantic metric Scene-Graph Intervention Accuracy (SGIA), which requires the full after-scene prediction to be correct and the only predicted before-to-after semantic change to be the intended object-factor edit. We also establish Delta-SGIA as a companion diagnostic that checks the single-site change pattern without requiring the full after-scene graph to be correct. Baseline evaluations on ground-truth-mask backbones, learned-slot models, SAM 2 + frozen-ViT models, and one mask-feature hybrid indicate that CoGenT-OOD-core degradation can persist under ground-truth instance masks, that mask source accounts for part but not all of native performance, and that locality or stability alone can overstate semantic faithfulness. Code is available at https://github.com/torux-bughunter/EditCLEVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。