用符号指导视觉规划,让模型更准地从图像中推理状态并做出正确动作。
Learning Visual Spatial Planning from Symbolic State via Modality-Gap-Aware Self-Distillation

- 通过两阶段自蒸馏,先用符号信息训练模型准确理解图像状态,再迁移规划能力。
- 在多个基准上提升性能19.3%和18.4%,显著缩小视觉输入与符号输入的差距。
- 训练时用符号,推理只依赖图像,适合实际部署的视觉决策场景。
尽管视觉语言模型在多模态理解上表现优异,但在视觉空间规划任务中仍存在困难。我们将其归因于感知与推理之间的模态鸿沟:视觉规划需从像素中推断潜在状态结构,并在此基础上进行合理决策,而符号规划则直接使用显式表示。这一差异带来两个连续瓶颈:感知阶段的状态恢复和推理阶段的多步规划。为此,我们提出MGSD——一种双阶段模态鸿沟感知的自蒸馏框架。第一阶段为冷启动接地阶段,在在线策略训练前建立可靠的视觉状态恢复能力;第二阶段为符号引导的在线策略自蒸馏阶段,通过学生生成前缀的词元级监督,将教师的规划行为传递给学生。关键在于,符号信息仅用于训练,推理阶段完全依赖视觉输入。在视觉规划基准上的实验表明,MGSD在不同模型规模下均显著提升性能,宏观平均分别提高19.3%和18.4%。消融与诊断分析进一步证实,性能提升源于视觉状态恢复与最优路径推理的双重改进。结果表明,MGSD不仅增强了从视觉观测中恢复可操作状态的能力,也提升了对推断结构的规划能力。代码已开源:https://github.com/Oranger-l/MGSD。
原文摘要 · Abstract (English)
While Vision-Language Models excel at general multimodal understanding, they still struggle with visual spatial planning. We attribute this limitation to a perception--reasoning modality gap. Visual planning requires models to infer latent state structures from pixels and then reason over the recovered structure to produce valid actions, whereas symbolic planning directly leverages explicit representation. This discrepancy introduces two sequential bottlenecks: visual state recovery at the perception stage and multi-step planning at the reasoning stage. To address this, we propose MGSD, a two-stage modality-gap-aware self-distillation framework. First, a cold-start grounding stage establishes reliable visual state recovery before on-policy training. Second, a symbol-guided on-policy self-distillation stage transfers the privileged teacher's planning behavior to the student through token-level supervision on student-generated prefixes. Crucially, symbolic information is used only during training, while inference relies exclusively on visual inputs. Experiments on visual planning benchmarks show that MGSD consistently improves performance across different model scales, raising the macro average by 19.3% and 18.4%, respectively. The resulting models substantially reduce the gap to the upper bounds obtained with symbolic inputs. Ablation studies and diagnostic analyses further confirm that the gains arise from improvements in both visual state recovery and optimal-path reasoning. These results demonstrate that MGSD strengthens not only the recovery of actionable states from visual observations but also the ability to plan over the inferred structures. Code is available at https://github.com/Oranger-l/MGSD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。