利用多层注意力融合提升草图语义分割精度,无需像素标注
LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation

- 通过聚合不同深度视觉变换器的注意力图,捕捉全局结构与局部笔画特征
- 在FS-COCO等数据集上相比基线模型提升3.43至15.74的平均交并比
- 适合需要弱监督、开放词汇的草图理解场景,如设计辅助与智能绘图
开放词汇草图语义分割旨在基于推理时指定的灵活类别词表,对稀疏线条图进行密集语义标注,且训练时不依赖像素级标注。与自然图像不同,草图缺乏纹理和颜色线索,语义理解高度依赖笔画布局与空间配置,导致单层视觉-语言特征不稳定。我们的关键观察是:不同层次的视觉变换器注意力图编码互补的空间线索——浅层捕捉全局结构布局,深层关注局部笔画交叉与物体部件。这表明跨层聚合能提供比单一层次更鲁棒的结构先验。基于此,我们提出结构感知框架LASA(Layer-wise Accumulated Structural Attention),通过多层注意力聚合,在弱监督下引导分层语义对齐,并在推理中优化预测。在FS-COCO、SFSD和FrISS上的实验表明,LASA相比先前弱监督基线,分别提升+3.43、+8.01和+15.74的mIoU,显著提升分割准确率与空间一致性。代码将公开。
原文摘要 · Abstract (English)
Open-vocabulary scene sketch semantic segmentation aims to assign dense semantic labels to sparse line drawings based on flexible category vocabularies specified at inference time, without relying on pixel-level annotations during training. Unlike natural images, sketches lack texture and color cues, making semantic understanding heavily dependent on stroke layout and spatial configuration, a challenge that renders single-layer vision-language features inherently unstable. Our key observation is that attention maps from different Vision Transformer layers encode complementary spatial cues: shallow layers capture global structural layouts, while deeper layers focus on local stroke intersections and object parts. This suggests that cross-layer aggregation provides a more robust structural prior than any individual layer alone. Leveraging this insight, we propose a structure-aware framework built upon \textbf{L}ayer-wise \textbf{A}ccumulated \textbf{S}tructural \textbf{A}ttention (\textbf{LASA}), which aggregates multi-layer attention to guide hierarchical semantic alignment under weak supervision and refine predictions during inference. Experiments on FS-COCO, SFSD, and FrISS show that LASA improves mIoU by $+3.43$, $+8.01$, and $+15.74$ over the prior weakly supervised baselines, demonstrating consistent gains in both segmentation accuracy and spatial coherence. Our source code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。