arXiv:2606.07639cs.CVcs.AI2026-06被引 2

提出双通道架构,让视频理解实时感知并动态修正答案。

MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention

论文配图:MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention
图 1 · 摘自论文原文
  • 用交叉注意力分离视觉与生成路径,实现感知不阻塞生成
  • 单卡处理256帧视频时首字延迟降低5倍,解码吞吐提升2.7倍
  • 适合需要实时交互的视频分析场景,如监控、人机对话

视频理解正从离线模式转向实时交互:模型在接收新帧的同时进行推理,根据新信息动态修正答案,并在无内容时保持沉默。本文提出MOSS-Video-Preview验证这一范式。核心观点是感知不应被生成阻塞,应采用双通道架构。相比主流解码器单体设计,交叉注意力骨干使视觉特征通过旁路输入,不进入自回归序列,实现感知与生成独立运行,减少视觉处理频率并提供清晰的通道接口用于独立压缩。我们构建数据合成管道,将密集描述转化为实时问答数据,其答案随模型已感知内容动态修正,并在此数据上微调离线模型以激发实时行为。尽管整体性能略逊于强基准Qwen2.5-VL-7B(主要归因于数据与规模),但模型在离线视频与多模态理解上表现良好,对空间与细粒度时间推理依然稳健,并具备离线模型缺乏的能力:持续感知、答案修订与适时沉默。在单张H200卡上处理256帧视频时,首字延迟约降低5倍,解码吞吐提升2.7倍,离线能力下降可忽略。本研究从范式、架构到数据,为实时视频理解提供了可行路径。

原文摘要 · Abstract (English)

Video understanding is shifting from the offline paradigm -- taking a fully recorded video as input and producing a single answer after it ends -- toward real-time interaction, in which the model perceives new frames while still replying, revises its answer as new evidence appears, and remains silent when there is nothing to say. We present MOSS-Video-Preview to validate this paradigm. Our central claim is that perception must not be blocked by generation; its natural realization is a two-channel architecture. We argue that a cross-attention backbone is better suited to real-time vision-language fusion than the prevailing decoder-only design: visual features enter through a side channel rather than joining the autoregressive sequence, so perception and generation run on separate, non-blocking pathways -- reducing the frequency of visual processing and exposing a clean channel-wise interface for independent compression. We complement this with a data synthesis pipeline that converts dense captions into real-time understanding QA whose answers are revised to match what the model has perceived so far, and we specialize an offline model on these data to elicit real-time behavior. Our model trails the strong Qwen2.5-VL-7B baseline overall -- a gap we attribute primarily to data and scale rather than the architecture -- yet attains competitive offline video and multimodal understanding, remains robust on the spatial and fine-grained temporal reasoning central to real-time use, and acquires behaviors that offline models lack: continuous perception, answer revision, and timely silence. On a single H200 with 256 frames per video, it achieves about a 5x speedup in time to first token and 2.7x higher decoding throughput, with negligible degradation in offline ability. Our study of paradigm, architecture, and data outlines a viable path toward real-time video understanding.

视频理解实时推理交叉注意力双通道

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。