用连续视觉表征替代离散输出，提升多模态推理效率与精度。
Latent Visual States for Efficient Multimodal Reasoning
- 提出潜空间槽令牌生成连续视觉表征，作为推理中间态。
- 在多个基准上实现显著性能提升，推理延迟降低。
- 适合需要高效多模态理解的系统开发者与研究者。
视觉证据的融合显著增强了大型多模态模型的能力。然而，当前方法主要依赖生成离散输出（如代码或框坐标）来调用外部工具，导致依赖关系僵化且延迟较高。为此，我们提出EVA（LatEnt Visual StAtes）框架，原生生成连续的潜在视觉表示。这些内部表示以自适应的潜空间槽令牌序列形式存在，作为推理过程中的中间视觉思维。潜空间槽令牌与离散文本令牌端到端联合训练，但这种联合优化导致潜空间与离散组件在‘过渡窗口’出现极端策略偏差。为此，我们提出D-GSPO（Decouple-GSPO）方法，通过解耦潜空间与离散组件的优化来解决根本问题。为支持监督微调，我们构建了EVA-230K数据集，一个高质量的图文交错思维链数据集，涵盖真实场景、文档、图表和OCR任务。大量实验表明，EVA在多个基准上均取得显著性能提升，同时提升了推理效率。
原文摘要 · Abstract (English)
The integration of visual evidence has significantly enhanced the capabilities of large multimodal models. However, this integration predominantly relies on generating discrete outputs (etc., code or box coordinates) to invoke external tools, a process that introduces rigid dependencies and substantial latency. To overcome these limitations, we propose {EVA} (LatEnt Visual StAtes), a novel framework that natively generates continuous latent visual representations. These internal representations manifest as an adaptive sequence of Latent\_slot tokens, serving as intermediate visual thoughts during the reasoning process. These Latent\_slot tokens are then trained end-to-end with the discrete text tokens. This co-optimization, notably, causes extreme policy deviation in the 'transition window' following the Latent\_slot tokens. We develop D-GSPO (Decouple-GSPO) to target this root cause by decoupling the optimization of latent and discrete components. To support SFT, we construct EVA-230K, a high-quality text-image interleaved CoT dataset encompassing a diverse range of real-world scenes, documents, charts and OCR tasks. Extensive experiments across multiple benchmarks confirm that EVA achieves significant performance gains while enhancing inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。