用多尺度体素特征增强语言模型输出,让3D物体交互区域定位更准。
VoxAfford: Multi-Scale Voxel-Token Fusion for Open-Vocabulary 3D Affordance Detection

- 生成文本后注入多尺度体素几何特征,提升空间感知能力。
- 在mIoU上比现有方法提升约8%,实现零样本迁移至新物体。
- 适合做开放词汇3D交互理解的科研与机器人应用。
开放词汇3D交互区域检测需在点云上定位给定新型交互描述的交互区域。现有方法通过将多模态大语言模型(MLLM)扩展为生成分割掩码的特殊输出标记,但这些标记采用自回归生成,建模的是序列依赖而非空间邻近关系,导致语义丰富但空间信息贫乏。本文提出体素增强的交互检测方法(VoxAfford),通过将冻结预训练3D VQVAE编码器提取的多尺度几何特征注入生成后的输出标记,解决此瓶颈。每个输出标记以其语义作为查询,通过交叉注意力从对应体素尺度中检索相关几何模式,并由学习的兼容性门控调节注入强度。增强后的标记通过语义条件注意力聚合为具有空间意识的交互提示,并与逐点特征一同传播以生成最终掩码。在开放词汇交互检测任务上的实验表明,VoxAfford达到当前最优性能,mIoU提升约8%;真实机器人实验验证了其对新物体的零样本迁移能力。
原文摘要 · Abstract (English)
Open-vocabulary 3D affordance detection requires localizing interaction regions on point clouds given novel affordance descriptions. Recent methods extend multimodal large language models (MLLMs) with special output tokens that are decoded into segmentation masks. However, these tokens are produced through autoregressive generation, which models sequential dependencies rather than spatial neighborhood relations, leaving them semantically rich but spatially impoverished for 3D localization. We propose Voxel-enhanced Affordance detection (VoxAfford), which bypasses this bottleneck by injecting multi-scale geometric features from a frozen pre-trained 3D VQVAE encoder into the output tokens after generation. Each output token uses its affordance semantics as a query to retrieve relevant geometric patterns from its paired voxel scale via cross-attention, with a learned compatibility gate controlling the injection strength. The enhanced tokens are then aggregated into a spatially-aware affordance prompt through semantic-conditioned attention and propagated alongside per-point features to generate the final mask. Experiments on open-vocabulary affordance detection tasks show that VoxAfford achieves state-of-the-art performance with approximately an 8% improvement in mIoU, and real robot experiments confirm zero-shot transfer to novel objects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。