用深度图和文本提示提升机器人功能感知,模型更小效果更好
Resource-Efficient Affordance Grounding with Complementary Depth and Semantic Prompts
- 用深度图当提示直接融入图像编码器,不加新参数
- 在AGD20K上比当前最好方法提升6.0%性能,参数减少88.8%
- 适合资源受限场景下的机器人功能理解任务
功能感知指智能体从环境中感知并利用的可用属性,是机器人执行动作的关键感知信息,具有丰富的多模态特性。现有方法因结构简单、融合方式基础且模型参数量大,难以满足实际部署的性能需求。为此,本文提出BiT-Align图像-深度-文本功能映射框架,包含旁路提示模块(BPM)和文本特征引导注意力选择机制(TFG)。BPM将辅助模态深度图作为提示直接嵌入主模态RGB图像编码器,无需引入额外编码器,降低参数量的同时显著提升功能区域定位精度。TFG利用文本特征引导图像编码器中注意力头的选择与增强,改善对功能特性的理解。实验表明,该方法在公开数据集AGD20K和HICO-IIF上均取得显著提升。在AGD20K上,相比当前最优方法,KLD指标提升6.0%,模型参数减少88.8%,具备实际应用价值。源代码将在https://github.com/DAWDSE/BiT-Align公开。
原文摘要 · Abstract (English)
Affordance refers to the functional properties that an agent perceives and utilizes from its environment, and is key perceptual information required for robots to perform actions. This information is rich and multimodal in nature. Existing multimodal affordance methods face limitations in extracting useful information, mainly due to simple structural designs, basic fusion methods, and large model parameters, making it difficult to meet the performance requirements for practical deployment. To address these issues, this paper proposes the BiT-Align image-depth-text affordance mapping framework. The framework includes a Bypass Prompt Module (BPM) and a Text Feature Guidance (TFG) attention selection mechanism. BPM integrates the auxiliary modality depth image directly as a prompt to the primary modality RGB image, embedding it into the primary modality encoder without introducing additional encoders. This reduces the model's parameter count and effectively improves functional region localization accuracy. The TFG mechanism guides the selection and enhancement of attention heads in the image encoder using textual features, improving the understanding of affordance characteristics. Experimental results demonstrate that the proposed method achieves significant performance improvements on public AGD20K and HICO-IIF datasets. On the AGD20K dataset, compared with the current state-of-the-art method, we achieve a 6.0% improvement in the KLD metric, while reducing model parameters by 88.8%, demonstrating practical application values. The source code will be made publicly available at https://github.com/DAWDSE/BiT-Align.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。