arXiv:2603.21986cs.CV2026-03被引 20

单流架构实现音视频生成速度与质量的突破,支持多语言真人内容生成。

Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model

  • 用单流Transformer统一处理文本、音视频,避免复杂交叉注意力设计。
  • 5秒256p视频仅需2秒生成,语音可懂率14.60%为开源模型最高。
  • 适合需要高效生成多语言真人音视频的开发者和创作者使用。

我们提出daVinci-MagiHuman,一个面向人类内容生成的开源音视频生成基础模型。该模型采用单流Transformer架构,通过自注意力机制统一处理文本、视频和音频的联合令牌序列,避免了多流或交叉注意力结构的复杂性,且兼容标准训练与推理基础设施。模型在人像生成场景中表现优异,能生成富有表现力的面部动作、自然的语音-表情协调、逼真的身体运动以及精确的音视频同步。支持中文(普通话与粤语)、英语、日语、韩语、德语和法语等多语言语音生成。为实现高效推理,结合模型蒸馏、潜空间超分辨率与Turbo VAE解码器,可在单张H100 GPU上实现5秒256p视频2秒生成。自动评估显示,其视觉质量与文本对齐度优于主流开源模型,语音可懂率最低达14.60%。人工成对对比测试中,相比Ovi 1.1胜率80.0%,相比LTX 2.3胜率60.9%(共2000次比较)。我们已开源完整模型栈,包括基础模型、蒸馏模型、超分模型及推理代码库。

原文摘要 · Abstract (English)

We present daVinci-MagiHuman, an open-source audio-video generative foundation model for human-centric generation. daVinci-MagiHuman jointly generates synchronized video and audio using a single-stream Transformer that processes text, video, and audio within a unified token sequence via self-attention only. This single-stream design avoids the complexity of multi-stream or cross-attention architectures while remaining easy to optimize with standard training and inference infrastructure. The model is particularly strong in human-centric scenarios, producing expressive facial performance, natural speech-expression coordination, realistic body motion, and precise audio-video synchronization. It supports multilingual spoken generation across Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French. For efficient inference, we combine the single-stream backbone with model distillation, latent-space super-resolution, and a Turbo VAE decoder, enabling generation of a 5-second 256p video in 2 seconds on a single H100 GPU. In automatic evaluation, daVinci-MagiHuman achieves the highest visual quality and text alignment among leading open models, along with the lowest word error rate (14.60%) for speech intelligibility. In pairwise human evaluation, it achieves win rates of 80.0% against Ovi 1.1 and 60.9% against LTX 2.3 over 2000 comparisons. We open-source the complete model stack, including the base model, the distilled model, the super-resolution model, and the inference codebase.

音视频生成单流架构多语言高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。