开源模型MOVA实现音视频同步生成，支持真实口型对齐与环境音效。
MOVA: Towards Scalable and Synchronized Video-Audio Generation
- 采用32B参数的MoE架构，推理时激活18B参数提升效率
- 支持图像+文本生成音视频，实现口型与语音精准同步
- 开源代码与权重，支持高效推理和LoRA微调，适合音视频创作者
音频是真实视频不可或缺的部分，但现有生成模型普遍忽视音频。当前音视频生成多依赖级联流程，导致成本高、误差累积且质量下降。尽管Veo 3和Sora 2等系统强调同步生成的重要性，但联合多模态建模在架构、数据和训练上仍面临独特挑战。此外，现有系统的闭源特性限制了领域发展。本文提出MOVA（MOSS Video and Audio），一个开源模型，可生成高质量、同步的音视频内容，包括真实口型对齐的语音、环境感知的声音效果及内容匹配的音乐。MOVA采用混合专家（MoE）架构，总参数量32B，推理时激活18B参数。支持图像-文本到音视频（IT2VA）生成任务。通过公开模型权重与代码，我们旨在推动研究并促进创作者社区发展。发布的代码库包含高效推理、LoRA微调和提示增强的完整支持。
原文摘要 · Abstract (English)
Audio is indispensable for real-world video, yet generation models have largely overlooked audio components. Current approaches to producing audio-visual content often rely on cascaded pipelines, which increase cost, accumulate errors, and degrade overall quality. While systems such as Veo 3 and Sora 2 emphasize the value of simultaneous generation, joint multimodal modeling introduces unique challenges in architecture, data, and training. Moreover, the closed-source nature of existing systems limits progress in the field. In this work, we introduce MOVA (MOSS Video and Audio), an open-source model capable of generating high-quality, synchronized audio-visual content, including realistic lip-synced speech, environment-aware sound effects, and content-aligned music. MOVA employs a Mixture-of-Experts (MoE) architecture, with a total of 32B parameters, of which 18B are active during inference. It supports IT2VA (Image-Text to Video-Audio) generation task. By releasing the model weights and code, we aim to advance research and foster a vibrant community of creators. The released codebase features comprehensive support for efficient inference, LoRA fine-tuning, and prompt enhancement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。