arXiv:2606.11837cs.CVcs.AI2026-06

利用多层注意力融合提升草图语义分割精度,无需像素标注

LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation

论文配图:LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation
图 1 · 摘自论文原文
  • 通过聚合不同深度视觉变换器的注意力图,捕捉全局结构与局部笔画特征
  • 在FS-COCO等数据集上相比基线模型提升3.43至15.74的平均交并比
  • 适合需要弱监督、开放词汇的草图理解场景,如设计辅助与智能绘图

开放词汇草图语义分割旨在基于推理时指定的灵活类别词表,对稀疏线条图进行密集语义标注,且训练时不依赖像素级标注。与自然图像不同,草图缺乏纹理和颜色线索,语义理解高度依赖笔画布局与空间配置,导致单层视觉-语言特征不稳定。我们的关键观察是:不同层次的视觉变换器注意力图编码互补的空间线索——浅层捕捉全局结构布局,深层关注局部笔画交叉与物体部件。这表明跨层聚合能提供比单一层次更鲁棒的结构先验。基于此,我们提出结构感知框架LASA(Layer-wise Accumulated Structural Attention),通过多层注意力聚合,在弱监督下引导分层语义对齐,并在推理中优化预测。在FS-COCO、SFSD和FrISS上的实验表明,LASA相比先前弱监督基线,分别提升+3.43、+8.01和+15.74的mIoU,显著提升分割准确率与空间一致性。代码将公开。

原文摘要 · Abstract (English)

Open-vocabulary scene sketch semantic segmentation aims to assign dense semantic labels to sparse line drawings based on flexible category vocabularies specified at inference time, without relying on pixel-level annotations during training. Unlike natural images, sketches lack texture and color cues, making semantic understanding heavily dependent on stroke layout and spatial configuration, a challenge that renders single-layer vision-language features inherently unstable. Our key observation is that attention maps from different Vision Transformer layers encode complementary spatial cues: shallow layers capture global structural layouts, while deeper layers focus on local stroke intersections and object parts. This suggests that cross-layer aggregation provides a more robust structural prior than any individual layer alone. Leveraging this insight, we propose a structure-aware framework built upon \textbf{L}ayer-wise \textbf{A}ccumulated \textbf{S}tructural \textbf{A}ttention (\textbf{LASA}), which aggregates multi-layer attention to guide hierarchical semantic alignment under weak supervision and refine predictions during inference. Experiments on FS-COCO, SFSD, and FrISS show that LASA improves mIoU by $+3.43$, $+8.01$, and $+15.74$ over the prior weakly supervised baselines, demonstrating consistent gains in both segmentation accuracy and spatial coherence. Our source code will be made publicly available.

草图分割弱监督视觉-语言结构先验

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。