arXiv:2606.25041cs.CVcs.AI2026-06被引 5

端到端实时交互模型,支持音视频同步响应。

Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

论文配图:Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
图 1 · 摘自论文原文
  • 统一模型处理多模态输入输出,用块因果注意力实现流式生成。
  • 模型响应延迟约200毫秒,总交互延迟约550毫秒。
  • 适合低延迟音视频交互场景,如虚拟助手、实时对话系统。

我们提出 Wan-Streamer,一种从零开始设计的原生流式、端到端交互式基础模型,专为实时、低延迟、全双工音视频交互而打造。Wan-Streamer 在单一 Transformer 中无缝建模语言、音频和视频作为输入与输出,序列由交错的视觉、音频和文本输入标记以及视觉、音频和文本输出标记组成,通过块因果注意力实现增量流式处理。不同于依赖独立语音活动检测(VAD)、自动语音识别(ASR)、语言模型、文本转语音(TTS)、音频驱动动画或视频生成模块的级联系统,Wan-Streamer 不依赖外部语言、语音、形象或视频生成模块:感知、推理、生成、响应时机、话轮管理及跨模态同步均在统一模型中联合学习,降低流水线延迟与误差累积。为支持自然的音视频响应,我们围绕可流式化重构整个技术栈,包括因果编码器、因果解码器、块因果注意力和低延迟多模态标记调度,实现每帧25 fps下最短160毫秒的流式单元。在结合350毫秒双向网络延迟时,模型侧响应延迟约为200毫秒,总交互延迟约为550毫秒,支持亚秒级全双工音视频通信。这些结果使 Wan-Streamer 成为面向低延迟流式交互的统一端到端多模态交互基础模型。

原文摘要 · Abstract (English)

We present Wan-Streamer, a native-streaming, end-to-end interactive foundation model designed from the ground up for real-time, low-latency, full-duplex audio-visual interaction. Wan-Streamer seamlessly models language, audio, and video as both input and output within a single Transformer, where the sequence is represented as interleaved visual, audio, and text input tokens together with visual, audio, and text output tokens, coordinated by block-causal attention for incremental streaming. Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules, Wan-Streamer does not rely on external language, speech, avatar, or video-generation modules: perception, reasoning, generation, response timing, turn management, and cross-modal synchronization are learned jointly within one unified model, reducing pipeline latency and error accumulation. To support natural audio-visual responsiveness, we redesign the entire stack around streamability, including causal encoders, causal decoders, block-causal attention, and low-latency multimodal token scheduling, enabling streaming units as short as 160 ms at 25 fps. Wan-Streamer achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency when combined with 350 ms bidirectional network latency, supporting sub-second duplex audio-visual communication. These results position Wan-Streamer as a unified, end-to-end, multimodal interactive foundation model for low-latency streaming interaction.

多模态交互流式生成实时响应基础模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。