让AI在3D场景中学会根据环境选工具,自动进化推理能力。
Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoning

- 构建场景记忆库,动态生成适配场景的智能工具使用策略。
- 工具使用率从39%提升至78%,关键任务准确率提高67%以上。
- 适合需要3D空间推理的智能体开发、机器人导航等场景。
本文研究代理型3D空间理解,即多模态大模型(MLLM)代理通过工具使用实现3D推理。现有方法常误用工具,在3D场景中表现出工具偏好偏差,使代理范式相比非代理策略仅带来微弱提升。我们发现3D空间推理任务在不同场景中具有异质性,而当前代理采用统一工具使用策略,未依据具体场景与任务选择工具。为此,我们提出Skill-3D框架,实现自演化场景感知技能。具体地,Skill-3D识别任务场景,将代理的工具使用轨迹记录至场景记忆中;相似场景的成功轨迹被聚合并提炼为可复用的场景感知技能,失败轨迹则作为教训附加。训练过程中,当相似场景再次出现,对应技能被注入以引导代理,新轨迹的成功与失败进一步优化技能,形成记忆与技能库共同演化的闭环。实验表明,Skill-3D显著提升3D空间推理中的工具利用率(在VSI-Bench上从39%升至78%),推动代理实现正确且充分的工具使用。例如,在MMSI-Bench上使Gemini-3-Flash性能提升67%。此外,基于技能引导轨迹进行代理后训练,使Qwen3-VL-8B在VSI-Bench上性能提升60%。
原文摘要 · Abstract (English)
This paper explores agentic 3D spatial understanding, i.e., MLLM agents performing 3D reasoning through tool use. Existing methods often misuse tools and exhibit biased tool preferences under 3D scenarios, leaving the agentic paradigm with only marginal gains over non-agentic strategies. We reveal that 3D spatial reasoning tasks are heterogeneous across scenes, while these agents apply a uniform tool-use strategy to all scenes rather than selecting tools according to the specific scene and task. To address this, we propose Skill-3D, a framework that learns self-evolving scene-aware skills. Specifically, Skill-3D identifies the task scene and records the agent's tool-use trajectory into a Scene Memory, where successful trajectories from similar scenes are aggregated and distilled into a reusable scene-aware skill, with failed ones attached to the skill as lessons. During training, once a similar scene recurs, the corresponding skill is injected to guide the agent, producing new trajectories whose successes and failures further refine the skill, forming a loop in which the memory and the skill library co-evolve. Experiments show that Skill-3D substantially improves tool utilization in 3D spatial reasoning (from 39% to 78% on VSI-Bench), driving the agent toward correct and sufficient tool use. For instance, it improves Gemini-3-Flash by 67% on MMSI-Bench. Furthermore, we conduct agentic post-training over skill-guided trajectories, which boosts Qwen3-VL-8B by 60% on VSI-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。