用剪枝+规划解决视觉语言模型探索时的来回晃动问题
Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answering
- 先用统计方法剔除不合理探索方向，再由覆盖优先的规划器做最终决策
- 在两个数据集上实现49%和33%的指标提升，相同探索预算下覆盖更广
- 适合需要稳定导航的智能体任务，尤其对过度自信的模型有明显改善
大型视觉语言模型（VLM）通过提供开放词汇推理的语义先验，提升了具身问答（EQA）智能体的表现。然而，直接用于步级探索时，VLM常出现前沿震荡，即因过度自信导致的不稳定往返移动，造成导航效率低下和答案质量下降。本文提出Prune-Then-Plan框架，通过步级校准稳定探索过程。不依赖原始VLM得分，而是采用受霍尔姆-邦费罗尼启发的剪枝机制剔除不合理前沿选择，再交由基于覆盖度的规划器做出最终决定。这种分离策略将过自信预测转化为保守且可解释的动作，借助人类水平判断校准VLM的步级行为。集成至3D-Mem EQA框架后，该方法在视觉接地SPL和LLM-Match指标上分别相比基线提升最高达49%和33%。整体上，在同等探索预算下，于OpenEQA与EXPRESS-Bench数据集均实现更优场景覆盖。
原文摘要 · Abstract (English)
Large vision-language models (VLMs) have improved embodied question answering (EQA) agents by providing strong semantic priors for open-vocabulary reasoning. However, when used directly for step-level exploration, VLMs often exhibit frontier oscillations, unstable back-and-forth movements caused by overconfidence and miscalibration, leading to inefficient navigation and degraded answer quality. We propose Prune-Then-Plan, a simple and effective framework that stabilizes exploration through step-level calibration. Instead of trusting raw VLM scores, our method prunes implausible frontier choices using a Holm-Bonferroni inspired pruning procedure and then delegates final decisions to a coverage-based planner. This separation converts overconfident predictions into conservative, interpretable actions by relying on human-level judgments to calibrate the step-level behavior of VLMs. Integrated into the 3D-Mem EQA framework, our approach achieves relative improvements of up to 49% and 33% in visually grounded SPL and LLM-Match metrics respectively over baselines. Overall, our method achieves better scene coverage under equal exploration budgets on both OpenEQA and EXPRESS-Bench datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。