让AI同时思考视觉信息的多种可能解释,更省资源地做多模态推理。
DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning

- 构建多个潜在视觉解释的连续隐空间,分别独立推理。
- 在多基准测试中准确率提升2-5点,内存消耗降低24%。
- 适合需要高效多视角推理的复杂视觉任务场景。
近年来多模态大语言模型的推理能力显著提升。现有方法通常依赖显式的思维链或连续隐空间轨迹来增强多步推理,但这些方法假设输入仅有一个隐式解释,并沿固定路径或统一计算预算展开推理。然而,在真实多模态场景中,视觉观察常受遮挡、模糊、视角变化或语义模糊影响,产生多种合理解释。统一推理策略不仅限制模型探索多种假设的能力,还导致高内存占用和推演成本。本文提出DLWM(Diverse Latent World Models),结合隐空间推理与强化学习的多模态推理框架。首先,在连续隐空间中构建一组多样化的潜在世界假设,每个假设捕捉视觉输入的不同合理解释,并对每个假设独立展开隐空间推理;通过正交性驱动的多样性正则化防止假设坍缩。其次,将隐空间推理建模为资源受限的序列决策问题,引入资源感知的强化学习策略,自适应分配计算资源于各假设,动态决定是否扩展、终止或合并推理路径,从而大幅降低内存开销并提升推演效率。在多个多模态推理基准测试中,DLWM相比现有方法准确率提升2-5个百分点,内存使用减少24%。
原文摘要 · Abstract (English)
Reasoning capabilities of multimodal large language models (MLLMs) have improved considerably in recent years. Existing approaches typically rely on explicit chain-of-thought or continuous latent-space trajectories to enhance multi-step reasoning. However, these methods generally assume that an input admits a single latent interpretation and unfold reasoning along a fixed path or under a uniform computation budget. In real-world multimodal settings, visual observations are often subject to occlusion, blur, viewpoint variation, or semantic ambiguity, giving rise to multiple plausible interpretations. A uniform reasoning strategy not only limits the model's ability to explore multiple hypotheses but also incurs high memory usage and rollout cost. We present DLWM (Diverse Latent World Models), a multimodal reasoning framework that combines latent-space reasoning with reinforcement learning. First, we construct a set of diverse latent world hypotheses in continuous latent space, each capturing a different plausible interpretation of the visual input, and unfold latent reasoning independently on each hypothesis. An orthogonality-based diversity regularizer explicitly prevents hypothesis collapse. Second, we formulate the latent reasoning process as a resource-constrained sequential decision problem and introduce a resource-aware reinforcement learning policy that adaptively allocates computation across hypotheses, dynamically deciding whether to expand, terminate, or merge reasoning paths, thereby substantially reducing memory footprint and improving rollout efficiency. Experiments on multiple multimodal reasoning benchmarks demonstrate that DLWM outperforms existing methods by 2-5 points in accuracy while reducing memory usage by 24%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。