提出Nava框架,实现音视频生成的原生对齐与可控音色控制。
Native Audio-Visual Alignment for Generation

- 在专用交互空间先对齐音视频,再用上下文条件联合去噪。
- 仅用6.3B参数即达到优秀画质、精准同步和强音色可控性。
- 适合需要高音视频一致性的生成任务,如虚拟人、配音应用。
联合音视频生成旨在合成时间同步且语义连贯的视听内容。现有开源方法多采用双塔结构后对齐,或统一三模态结构混合文本、音频与视频,前者削弱音视频细粒度协同演化,后者将语义条件与底层同步耦合。为此,我们提出NAVA,一种原生音视频对齐框架。NAVA基于上下文感知的原生对齐:先在专用交互空间建立音视频对应关系,再用外部上下文引导联合去噪过程。具体地,采用Align-then-Fuse MMDiT架构,从模态感知对齐过渡到模态共享联合去噪。此外,引入“上下文中的音色条件”机制,将参考音色线索关联至对应语音片段,实现可控语音音色。在Verse-Bench和Seed-TTS上的实验及用户研究显示,NAVA仅用6.3B参数即可实现优异视频质量、精确音视频同步、竞争性音频质量以及更强的参考音色可控性。
原文摘要 · Abstract (English)
Joint audio-video generation aims to synthesize temporally synchronized and semantically coherent visual-acoustic content. However, existing open-source methods mainly rely on either dual-tower designs with posterior alignment or fully unified tri-modal designs that mix textual context, audio and video in one shared space. The former weakens fine-grained audio-video co-evolution, while the latter couples semantic conditioning with low-level synchronization. To address these limitations, we propose NAVA, a Native Audio-Visual Alignment framework for joint audio-video generation. NAVA is built upon context-conditioned native audio-visual alignment: it first establishes audio-video correspondence in a dedicated interaction space, and then uses external context to condition the joint denoising process. Specifically, NAVA is instantiated with an Align-then-Fuse MMDiT architecture, which transitions from modality-aware audio-video alignment to modality-shared joint denoising. Furthermore, we introduce Timbre-in-Context Conditioning to associate reference timbre cues with corresponding speech spans to achieve controllable speech timbre. Experiments on Verse-Bench and Seed-TTS, together with a user study, demonstrate that NAVA achieves superior video quality, precise audio-visual synchronization, competitive audio quality, and stronger reference-timbre controllability using only 6.3B parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。