提升自监督模型对物体属性的理解能力,让机器更准确识别具体物体。
Oh-A-DINO: Understanding and Enhancing Attribute-Level Information in Self-Supervised Object-Centric Representations
- 在分割图像块上学习隐变量空间,用变分自编码器约束表示的紧凑性和解耦性。
- 自监督模型能识别形状大小但丢失颜色材质等表面属性,新方法补全这些缺失信息。
- 适合需要精细物体理解的任务,如目标检索、机器人操作和基于目标的策略执行。
物体中心理解是人类视觉的基础,也是复杂推理所必需的。传统方法通过槽位瓶颈显式学习物体属性,而最近的自监督视觉模型(如DINO)展现出涌现的物体理解能力。我们评估了CLIP、DINOv2和DINOv3等自监督模型及槽位方法在多物体实例检索任务中的表现,该任务要求在场景中精准识别特定物体。随着预训练表示被广泛用于下游任务(如检索、操控和目标条件策略),这一需求日益重要。研究发现,自监督模型和槽位表示擅长捕捉由边缘推导出的几何属性(形状、尺寸),但无法保留非几何的表面线索(颜色、材质、纹理),这些线索在推理或选择物体时至关重要。我们提出在分割图像块上学习一个辅助隐变量空间,通过变分自编码器正则化,使表示更紧凑且解耦。将该隐变量引入自监督模型后,所有属性的检索性能均得到提升,表明该方向有助于增强自监督表示在需要精确物体级推理任务中的可靠性。
原文摘要 · Abstract (English)
Object-centric understanding is fundamental to human vision and required for complex reasoning. Traditional methods define slot-based bottlenecks to learn object properties explicitly, while recent self-supervised vision models like DINO have shown emergent object understanding. We investigate the effectiveness of self-supervised representations from models such as CLIP, DINOv2 and DINOv3, as well as slot-based approaches, for multi-object instance retrieval, where specific objects must be faithfully identified in a scene. This scenario is increasingly relevant as pre-trained representations are deployed in downstream tasks, e.g., retrieval, manipulation, and goal-conditioned policies that demand fine-grained object understanding. Our findings reveal that self-supervised vision models and slot-based representations excel at identifying edge-derived geometry (shape, size) but fail to preserve non-geometric surface-level cues (colour, material, texture), which are critical for disambiguating objects when reasoning about or selecting them in such tasks. We show that learning an auxiliary latent space over segmented patches, where VAE regularisation enforces compact, disentangled object-centric representations, recovers these missing attributes. Augmenting the self-supervised methods with such latents improves retrieval across all attributes, suggesting a promising direction for making self-supervised representations more reliable in downstream tasks that require precise object-level reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。