arXiv:2605.22012cs.CLcs.CV2026-05被引 2

用统一潜在空间提升音视频联合推理能力,更精准捕捉细节。

LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning

论文配图:LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
图 1 · 摘自论文原文
  • 将文本推理与音视频潜在状态交替进行,保留感官信息密度
  • 在多个基准上优于开源模型,显著超越传统文本链式推理
  • 适合需要精细音视频理解的多模态研究者

联合音视频推理对全模态理解至关重要,但现有多模态大模型在需跨模态细粒度证据时仍表现不佳。核心瓶颈在于显式的文本链式思维(CoT)将连续音视频信号压缩为离散标记,削弱时间定位并使中间推理偏向语言先验。本文提出LatentOmni框架,主张统一潜在空间更适合此类推理——既保留密集感官信息,又兼容自回归生成。该框架通过特征级监督对齐潜在推理状态与任务相关感官特征,并引入Omni-Sync位置嵌入(OSPE)保持音视频潜在状态的时间一致性。我们进一步构建了包含35,000条音视频交错推理轨迹的LatentOmni-Instruct-35K数据集,用于监督潜在空间推理。在多个音视频推理基准上的综合评估表明,LatentOmni在所评测开源模型中表现最佳,持续优于显式文本链式思维基线,验证了潜在空间联合推理作为强全模态理解路径的潜力。

原文摘要 · Abstract (English)

Joint audio-visual reasoning is essential for omnimodal understanding, yet current multimodal large language models (MLLMs) still struggle when reasoning requires fine-grained evidence from both modalities. A central limitation is that explicit text-based chain-of-thought (CoT) compresses continuous audio-visual signals into discrete tokens, weakening temporal grounding and shifting intermediate reasoning toward language priors. We argue that a unified latent space is a better medium for such reasoning because it preserves dense sensory information while remaining compatible with autoregressive generation. Based on this insight, we propose \textbf{LatentOmni}, a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states. LatentOmni introduces feature-level supervision to align latent reasoning states with task-relevant sensory features and uses Omni-Sync Position Embedding (OSPE) to maintain temporal consistency between latent audio and visual states. We further construct \textbf{LatentOmni-Instruct-35K}, a dataset of audio-visual interleaved reasoning trajectories for supervising latent-space reasoning. Comprehensive evaluation across multiple audio-visual reasoning benchmarks demonstrates that LatentOmni achieves the best performance among the evaluated open-source models and consistently outperforms the Explicit Text CoT baseline, supporting latent-space joint reasoning as a promising path toward stronger omnimodal understanding.

多模态潜在空间音视频理解推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。