用逻辑规则约束像素关系,让图像分割更懂物体位置关系。
Neuro-Symbolic Spatial Reasoning in Segmentation
- 将空间关系转化为一阶逻辑公式,通过伪类别编码像素位置约束。
- 在四个数据集上平均交并比达最新水平,多类别场景下优势明显。
- 仅加一个辅助损失,零额外参数,适合追求高精度分割的开发者。
开放词汇语义分割(OVSS)需对未见、未标注物体进行像素级分类,传统基于视觉语言模型的方法缺乏对场景中物体空间关系的理解。为此,本文提出首个在OVSS中应用神经符号(NeSy)空间推理的方法——关联分割器(RelateSeg)。该方法首先自动提取空间关系(如“猫在人右侧”),用提出的伪类别编码为一阶逻辑公式,并让每个像素同时预测语义类别(如“猫”)和空间伪类别(如“人右侧”),强制满足空间约束(如“猫”像素必须位于“人”右侧)。通过模糊逻辑松弛将这些逻辑约束嵌入深度网络,实现端到端学习。实验表明,RelateSeg在四个基准数据集上达到最高平均mIoU,尤其在多类别图像中表现突出,仅引入单一辅助损失且不增加额外参数,验证了神经符号空间推理的有效性。
原文摘要 · Abstract (English)
Open-Vocabulary Semantic Segmentation (OVSS) assigns pixel-level labels from an open set of categories, requiring generalization to unseen and unlabelled objects. Using vision-language models (VLMs) to correlate local image patches with potential unseen object categories suffers from a lack of understanding of spatial relations of objects in a scene. To solve this problem, we introduce neuro-symbolic (NeSy) spatial reasoning in OVSS. In contrast to contemporary VLM correlation-based approaches, we propose Relational Segmentor (RelateSeg) to impose explicit spatial relational constraints by first order logic (FOL) formulated in a neural network architecture. This is the first attempt to explore NeSy spatial reasoning in OVSS. Specifically, RelateSeg automatically extracts spatial relations, e.g., <cat, to-right-of, person>, and encodes them as first-order logic formulas using our proposed pseudo categories. Each pixel learns to predict both a semantic category (e.g., "cat") and a spatial pseudo category (e.g., "right of person") simultaneously, enforcing relational constraints (e.g., a "cat" pixel must lie to the right of a "person"). Finally, these logic constraints are formulated in a deep network architecture by fuzzy logic relaxation, enabling end-to-end learning of spatial-relationally consistent segmentation. RelateSeg achieves state-of-the-art performance in terms of average mIoU across four benchmark datasets and particularly shows clear advantages on images containing multiple categories, with the cost of only introducing a single auxiliary loss function and no additional parameters, validating the effectiveness of NeSy spatial reasoning in OVSS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。