让视觉语言模型在隐空间学用工具，推理更快更准
Multimodal Latent Reasoning via Predictive Embeddings
- 在隐空间直接学预测嵌入，无需推理时调用工具
- 多任务基准测试表现超监督微调和传统生成方法
- 适合追求高效、稳定工具调用的视觉推理场景
工具增强的多模态推理使视觉语言模型（VLMs）通过与外部工具（如裁剪、深度估计）交互提升感知能力。然而，现有方法存在显著推理开销、需特殊标注且易误调工具。我们提出Pearl（Predictive Embedding Alignment for Reasoning in Latent space），一种受JEPA启发的框架，在隐空间完全学习专家工具使用轨迹，推理时无需显式调用工具。不同于基于重建的隐空间推理方法（需自回归生成隐状态，存在训练-推理不一致且难支持多步操作），Pearl直接从多模态轨迹中学习预测嵌入，保持标准视觉-语言生成流程：模型无关、训练简单，并自然支持多步工具调用。多个感知基准实验表明，Pearl在性能上达到或超越监督微调与重建类方法。此外，实证发现重建类方法主要学习嵌入而非图像编辑，支持预测嵌入学习作为更合理替代方案。
原文摘要 · Abstract (English)
Tool-augmented multimodal reasoning enables visual language models (VLMs) to improve perception by interacting with external tools (e.g., cropping, depth estimation). However, such approaches incur substantial inference overhead, require specialized supervision, and are prone to erroneous tool calls. We propose Pearl (Predictive Embedding Alignment for Reasoning in Latent space), a JEPA-inspired framework that learns from expert tool-use trajectories entirely in the latent space, eliminating the need for explicit tool invocation at inference time. Unlike reconstruction-based latent reasoning methods, which autoregressively generate latent tokens and suffer from training-inference mismatch and limited support for multi-step tool use, Pearl directly learns predictive embeddings from multimodal trajectories while preserving the standard vision-language generation pipeline: it is model-agnostic, simple to train, and naturally supports trajectories with multiple tool calls. Experiments across multiple perception benchmarks show that Pearl matches or outperforms standard supervised fine-tuning and reconstruction-based latent reasoning approaches. Furthermore, we provide empirical evidence that reconstruction-based methods primarily learn embeddings rather than image edits in latent space, motivating predictive embedding learning as a more principled alternative.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。