用部件语义先验引导弱监督抓取定位,提升模型对物体功能的理解能力。
Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors
- 基于预训练部件分割模型生成伪标签,结合功能到部件的映射关系。
- 在多个数据集上显著超越现有方法,实现突破性性能提升。
- 适合研究视觉-动作关联、弱监督学习与具身智能的开发者参考。
本文研究弱监督抓取定位任务,即在无密集标注的情况下,利用人-物交互图像和第一人称物体图像识别物体上的功能区域。以往方法多依赖类激活图,适用于语义分割但难以精准定位动作与功能。我们借助先进的基础模型,构建基于伪标签的监督训练流程:伪标签由现成的部件分割模型生成,并通过功能名称到部件名称的映射进行引导。此外,提出三项关键改进:标签精炼阶段、细粒度特征对齐过程和轻量级推理模块。这些技术利用基础模型中嵌入的静态物体语义知识,有效弥合物体与动作之间的鸿沟。大量实验表明,所提模型性能显著超越现有方法。代码已开源:https://github.com/woyut/WSAG-PLSP。
原文摘要 · Abstract (English)
In this work, we focus on the task of weakly supervised affordance grounding, where a model is trained to identify affordance regions on objects using human-object interaction images and egocentric object images without dense labels. Previous works are mostly built upon class activation maps, which are effective for semantic segmentation but may not be suitable for locating actions and functions. Leveraging recent advanced foundation models, we develop a supervised training pipeline based on pseudo labels. The pseudo labels are generated from an off-the-shelf part segmentation model, guided by a mapping from affordance to part names. Furthermore, we introduce three key enhancements to the baseline model: a label refining stage, a fine-grained feature alignment process, and a lightweight reasoning module. These techniques harness the semantic knowledge of static objects embedded in off-the-shelf foundation models to improve affordance learning, effectively bridging the gap between objects and actions. Extensive experiments demonstrate that the performance of the proposed model has achieved a breakthrough improvement over existing methods. Our codes are available at https://github.com/woyut/WSAG-PLSP .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。