用统一模型同时生成语音和人脸动作，让说话更自然。
JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching
- 用流匹配+多模态扩散变换器，联合建模语音与面部动作。
- 支持文本、音频或动作参考输入，生成同步的说话头像。
- 适合做视频生成、语音驱动动画等跨模态应用。
语音与面部动作之间的内在联系在生成建模中常被忽视，当前说话头生成与文本转语音通常作为独立任务处理。本文提出 JAM-Flow，一个统一框架，可同时合成并条件化面部运动与语音。该方法采用流匹配技术，结合新型多模态扩散变换器（MM-DiT）架构，包含专用的运动-迪特（Motion-DiT）与音频-迪特（Audio-DiT）模块，通过选择性联合注意力层连接，并引入时间对齐位置编码与局部联合注意力掩码，以实现有效跨模态交互，同时保留各模态特性。模型以填入式目标进行训练，支持多种条件输入——包括文本、参考音频与参考动作，适用于从文本生成同步说话头像、音频驱动动画等多种任务。JAM-Flow 为端到端音视同步生成提供了实用解决方案，显著推动多模态生成建模发展。
原文摘要 · Abstract (English)
The intrinsic link between facial motion and speech is often overlooked in generative modeling, where talking head synthesis and text-to-speech (TTS) are typically addressed as separate tasks. This paper introduces JAM-Flow, a unified framework to simultaneously synthesize and condition on both facial motion and speech. Our approach leverages flow matching and a novel Multi-Modal Diffusion Transformer (MM-DiT) architecture, integrating specialized Motion-DiT and Audio-DiT modules. These are coupled via selective joint attention layers and incorporate key architectural choices, such as temporally aligned positional embeddings and localized joint attention masking, to enable effective cross-modal interaction while preserving modality-specific strengths. Trained with an inpainting-style objective, JAM-Flow supports a wide array of conditioning inputs-including text, reference audio, and reference motion-facilitating tasks such as synchronized talking head generation from text, audio-driven animation, and much more, within a single, coherent model. JAM-Flow significantly advances multi-modal generative modeling by providing a practical solution for holistic audio-visual synthesis. project page: https://joonghyuk.com/jamflow-web
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。