arXiv:2605.25195cs.CV2026-05被引 2

用显式语义蓝图统一音视频生成,解决同步与细节难题

Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation

论文配图:Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation
图 1 · 摘自论文原文
  • 引入多模态规划器生成音视频关键帧语义蓝图
  • 生成的蓝图使音视频同步率提升18.6%,细节更丰富
  • 适合需要高精度跨模态对齐的研究者与开发者

当前开源扩散模型在复杂语义推理场景下难以生成稳定同步的音视频内容。根源在于现有方法依赖通用编码器的粗粒度文本嵌入引导音视频去噪,丢失细粒度语义且缺乏共享的长时程规划,导致去噪轨迹不协调、跨模态对齐脆弱。本文提出Baton,首个将显式语义规划引入联合音视频生成的框架。核心思想是:在粗粒度文本引导之外,加入语义丰富、模态感知的规划标记,于去噪前进行联合推理与相互对齐,以同时恢复细粒度语义并建立共享蓝图协调音视频去噪路径。具体而言,Baton设计了带有双语义对齐塔的VA-Planner,可学习查询交叉关注音视频特征,生成一对语义对齐的音视频规划标记作为关键帧级蓝图。这些标记通过交叉注意力注入扩散主干,提供时间定位的补充引导。由于规划标记与扩散潜变量无一一对应关系,我们进一步提出相对语义RoPE,将二者映射至共享时空坐标系,使每个潜变量能准确关联其对应语义线索。在基准测试上,Baton在定性与定量上均验证了有效性。

原文摘要 · Abstract (English)

Current open-source diffusion models struggle to generate stable and synchronized audio-visual content, particularly in scenarios demanding complex semantic reasoning. The root cause is that existing methods rely on coarse text embeddings from off-the-shelf encoders to guide audio-video denoising, which discards fine-grained semantics and, critically, lacks a shared long-horizon plan, leading to uncoordinated denoising trajectories and fragile cross-modal alignment. We propose Baton, the first framework that introduces explicit semantic planning into joint video-audio generation. Our key insight is that complementing coarse text guidance with semantically rich, modality-aware planned tokens, jointly reasoned and mutually aligned before denoising, can simultaneously restore fine-grained semantic detail and establish a shared blueprint that coordinates both audio and video denoising trajectories. Concretely, Baton first introduces the VA-Planner, a multimodal language model equipped with dual semantic alignment towers, where learnable queries cross-attend to both video and audio features to produce a pair of semantically aligned video and audio planned tokens as keyframe-level blueprints. These planned tokens are injected into the diffusion backbone via cross-attention layers, providing temporally grounded guidance complementary to coarse text embeddings. Since planned tokens do not share one-to-one spatial-temporal correspondence with diffusion latents, we further propose Relative Semantic RoPE, a relative positional encoding that maps planned tokens and latents into a shared spatial-temporal coordinate frame, enabling each latent to accurately attend to its positionally corresponding semantic cues. Experiments on benchmarks show the effectiveness of Baton both qualitatively and quantitatively.

音视频生成扩散模型语义规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。