arXiv:2608.15045cs.CV2026-08

MOSS-VL让视觉语言模型实时互动,边看边说更自然。

MOSS-VL Technical Report

  • 视觉输入通过门控交叉注意力动态接入,实现边看边生成。
  • 在四个流式评测中三项领先,主动预警任务准确率66.0%远超基线。
  • 适合需要低延迟实时交互的应用,如智能助手、自动驾驶感知。

我们提出MOSS-VL,一个将实时交互(边感知边生成)作为核心能力的开源多模态模型家族。其语言解码器通过门控交叉注意力仅在需要时关注视觉信息,支持在生成过程中自然接收新帧。训练采用合成交互语料库,指导模型何时说话、沉默或修正;分阶段课程设计将实时训练集中在强预训练模型之后的轻量级最终阶段。离线模式下,MOSS-VL-Instruct在同等规模下表现优异,尤其在时间推理视频任务上领先。在四个流式基准测试中,MOSS-VL-Realtime在三个上取得最高平均分(第四个第二),在直接考察主动行为的三个子集上全面胜出——主动预警任务达66.0%,优于最佳基线37.5%。尽管参数量为11.3B,但因视觉令牌不参与解码序列,随视觉上下文增长,其首次生成时间优势从2.8倍扩大至5.1倍,优于同架构的Qwen3-VL-8B。我们已开放全部五个检查点、训练流程和实时推理代码,详见https://github.com/OpenMOSS/MOSS-VL。

原文摘要 · Abstract (English)

We present MOSS-VL, an open vision-language model family that treats real-time interaction -- perceiving while it speaks -- as a first-class capability. It is co-designed across the stack: the language decoder attends to vision only through gated cross-attention, so the model can naturally see incoming frames while generating; a synthesized interaction corpus supervises when to speak, when to stay silent, and when to revise; and a staged curriculum concentrates all real-time-specific training in one light final stage over a strong offline foundation. Offline, MOSS-VL-Instruct is competitive at comparable scale and leads temporal-reasoning video sets. Across four streaming benchmarks, MOSS-VL-Realtime posts the best average on three (second on the fourth) among open-source streaming models, sweeping the three subsets that squarely test proactive behavior -- 66.0 vs. 37.5 for the best baseline on OmniMMI Proactive Alerting. With 11.3B parameters but visual tokens outside the decoded sequence, MOSS-VL widens its time-to-first-token advantage over same-backbone Qwen3-VL-8B from 2.8x to 5.1x as visual context grows. We release all five checkpoints, the training curriculum, and the real-time inference code at https://github.com/OpenMOSS/MOSS-VL.

视觉语言模型实时交互多模态生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。