arXiv:2605.01506cs.CV2026-05

统一编码视觉与音频信号,实现人类般的连续运动感知。

OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder

论文配图:OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder
图 1 · 摘自论文原文
  • 采用统一的25帧/秒编码架构,同步处理视觉与音频信号。
  • 在手语识别和精细运动分析任务上显著超越基线模型。
  • 适合需要跨模态连续理解的多模态应用开发。

近期的多模态大语言模型进展推动了视觉-音频联合理解能力的提升。然而,现有架构普遍采用特定模态编码器,遵循‘视频粗粒度、音频细粒度’的设计——视频帧以1–2帧/秒采样,音频波形以25帧/秒处理——导致模型以‘逐帧、逐模态’方式感知,而非像人类一样整体化理解。这种差异使模型在编码阶段缺乏充分的跨模态交互,难以捕捉细微视觉运动。为此,我们提出**Omni-Encoder**,一种统一的Transformer主干网络,能够在共享潜在空间中以对称的25帧/秒速率联合嵌入视觉与音频信号。该架构通过三项核心创新——Omni-Encoder Token Template、Omni-RoPE与Temporal Window Shifting——有效应对模态解耦与计算效率的双重挑战。实验表明,在相同输入令牌预算下,相比基线模型Qwen2.5-Omni,Omni-Encoder在视觉连续理解任务(如手语识别、精细体育动作分析)上取得显著提升,同时在AVQA和说话人识别定位等经典音频-视觉基准测试中保持竞争力。结果表明,统一的全知式编码为构建更贴近人类感知整合特性的多模态模型提供了可行路径。

原文摘要 · Abstract (English)

Recent advances in omni-modal large language models have enabled remarkable progress in joint vision-audio understanding. However, prevailing architectures rely on modality-specific encoders with a \emph{video-coarse, audio-dense} design -- sampling visual frames at 1--2 fps while processing audio waveforms at 25 fps -- resulting in systems that perceive video \emph{frame by frame, modality by modality} rather than holistically as humans do. Such a discrepancy leaves models with impoverished cross-modal interaction during encoding and an inability to capture fine-grained visual motion. To bridge this gap, we present \textbf{Omni-Encoder, a unified Transformer backbone designed to co-embed visual and audio signals at a symmetrical 25 fps} within a shared latent space. This architecture leverages three core innovations -- the Omni-Encoder Token Template, Omni-RoPE, and Temporal Window Shifting -- to effectively reconcile the dual challenges of modality disentanglement and computational efficiency. Experiments demonstrate that, compared to the modality-specific baseline Qwen2.5-Omni under the same input token budget to the LLM decoder, Omni-Encoder delivers substantial gains on visual continuous understanding tasks -- such as sign language recognition and fine-grained sports action analysis -- while maintaining competitive performance on established audio-visual benchmarks such as AVQA and Speaker Identification and Localization. These results suggest that unified omnivorous encoding offers a promising direction for building omni-modal models that more closely reflect the integrated nature of human perception.

多模态连续理解统一编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。