arXiv:2608.08469cs.AI2026-08

实现输入输出同步的实时多模态生成,让模型像对话一样即时响应。

Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation

论文配图:Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation
图 1 · 摘自论文原文
  • 输入输出按80毫秒时间片对齐,实现双向实时生成
  • 20分钟流式视频处理中位延迟仅84毫秒,95%分位173毫秒
  • 适合需要低延迟交互的实时语音视频应用

现有流式多模态模型虽增量处理观测,但仍采用逐轮预填充-解码模式,无法实现真正的双向交互。主动式替代方案使用微轮询或外部响应门控,破坏连续性,使响应时机与生成脱节,并增加KV缓存友好部署难度。本文提出Aero Realtime,一个40亿参数的流式多模态模型,采用全双工架构实现实时生成。该模型将视频、音频和文本输出对齐至统一的时间网格,每个约80毫秒的音频槽预测词汇标记或静音标记。这使得输入与输出可同步推进,单一自回归目标同时学习响应时机与内容生成。推理时,仅追加最新多模态槽,延续前序输出状态,并复用KV缓存以实现高效增量执行。我们还提供了完整的训练与服务方案,包括实时问答构建、槽对齐监督、硬件感知分布式训练及可恢复推理。在四块NVIDIA A6000工作站显卡上,Aero Realtime在持续20分钟的视频流上保持84毫秒中位处理延迟和173毫秒P95延迟,始终维持在源时间线200毫秒以内。结果证明了全对齐输入输出建模在双工、主动与硬件适配的多模态交互中的可行性。

原文摘要 · Abstract (English)

Existing streaming multimodal models process observations incrementally but still follow a turn-based prefill-then-decode pattern, making them non-duplex: new observations cannot naturally enter an active generation stream. Proactive alternatives use micro-turn polling or external response gates, which fragment continuous interaction, decouple response timing from language generation, and complicate KV-cache-friendly serving. We introduce Aero Realtime, a 4B streaming multimodal model with a duplex architecture for realtime generation. Aero Realtime aligns video, audio, and textual output on a shared temporal grid, where each approximately 80-ms audio slot predicts either a lexical token or a silence token. This allows input and output to advance together, enabling one autoregressive objective to learn both when to respond and what to generate. During inference, Aero Realtime appends only the newest multimodal slot, carries forward the previous output state, and reuses the KV cache for efficient incremental execution. We further provide a complete training and serving recipe, including realtime QA construction, slot-aligned supervision, hardware-aware distributed training, and resumable inference. On four NVIDIA A6000 workstation GPUs, Aero Realtime maintains 84-ms median and 173-ms P95 processing lag over 20 minutes of a continuously streamed video, remaining within 200~ms of the source timeline. These results demonstrate the feasibility of fully aligned input-output modeling for duplex, proactive, and hardware-aligned multimodal interaction.

实时生成多模态低延迟流式处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。