根据问题自动选最佳模态，让机器人更准回答3D空间问题。
MASER: Modality-Adaptive Specialist Routing for Embodied 3D Spatial Intelligence
- 用神经路由动态选择最适合问题的模态适配器。
- 在Open3D-VQA上点云模态最优，占比51.5%；路由准确率达51.3%。
- 轻量设计，每题仅调用一次适配器，适合实际部署。
在3D环境中，具身智能体需通过自然语言、RGB图像、点云、深度图和相机位姿等多种模态推理来回答空间相关问题。现有视觉-语言模型（VLM）通常仅在单一模态上微调，忽略问题语义可能偏好不同模态的事实。为此，我们提出MASER（Modality-Adaptive SpEcialist Routing），一个轻量级框架：在共享的VLM主干上训练五个不同模态适配器，并学习一个神经路由策略，在推理时根据问题选择最优适配器。每个问题通过冻结的句子变换器编码，其嵌入经由一个小型MLP（基于原始适配器准确率标签训练）生成路由决策。在Open3D-VQA基准上的评估表明，无单一模态始终最优——点云在51.5%的情况下表现最佳。MASER路由与理想路由达成51.3%的一致性，优于随机森林基线（43.5%），且每题仅需调用一次适配器。
原文摘要 · Abstract (English)
In 3D environments, Embodied Agents answer spatially relevant questions through reasoning from a mixture of modalities including natural language, RGB images, point clouds, depth maps and camera poses. Existing Vision-Language models (VLMs) are fine-tuned over a single modality. This completely ignores the question semantics which may favor a different modality than the finetuned modality. To address this, we propose MASER (Modality-Adaptive SpEcialist Routing), a lightweight framework that trains five different modality adapters of a shared VLM backbone and learns a neural routing policy that selects the best adapter based on the question during inference. We encode each question with a frozen sentence transformer and pass the embedding through a small Multi-layer Perceptron (MLP) trained on oracle adapter-accuracy labels. We evaluate our methodology over the Open3D-VQA benchmark and our evaluations show that no single modality is universally optimal -- point-cloud answers are best in 51.5% of cases. MASER routes with 51.3% oracle agreement, outperforming a Random-Forest ablation (43.5%), with only a single adapter call per question.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。