用音频实时驱动虚拟形象，实现自然对话体验
TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models
- 将图像转视频模型改造成音频驱动的真人级动画生成器
- 180亿参数模型支持无误差无限时长视频流生成
- 通过分设备计算与通信重叠优化，实现低延迟高吞吐
本文提出TalkingMachines——一个高效框架，将预训练视频生成模型转化为实时、音频驱动的角色动画系统。该系统通过整合音频大语言模型与视频生成基础模型，实现自然对话体验。主要贡献包括：（1）将SOTA图像转视频DiT模型适配为180亿参数的音频驱动角色生成模型；（2）通过双向教师模型向稀疏因果自回归学生模型进行非对称知识蒸馏，实现无误差积累的无限视频流生成；（3）设计高吞吐、低延迟推理管道，包含多项工程优化：（a）将DiT与VAE解码器部署在不同设备上；（b）利用CUDA流实现设备间通信与计算重叠；（c）消除冗余重计算以最大化帧生成吞吐量。演示视频见https://aaxwaz.github.io/TalkingMachines/
原文摘要 · Abstract (English)
In this paper, we present TalkingMachines -- an efficient framework that transforms pretrained video generation models into real-time, audio-driven character animators. TalkingMachines enables natural conversational experiences by integrating an audio large language model (LLM) with our video generation foundation model. Our primary contributions include: (1) We adapt a pretrained SOTA image-to-video DiT into an audio-driven avatar generation model of 18 billion parameters; (2) We enable infinite video streaming without error accumulation through asymmetric knowledge distillation from a bidirectional teacher model into a sparse causal, autoregressive student model; (3) We design a high-throughput, low-latency inference pipeline incorporating several key engineering optimizations such as: (a) disaggregation of the DiT and VAE decoder across separate devices, (b) efficient overlap of inter-device communication and computation using CUDA streams, (c) elimination of redundant recomputations to maximize frame-generation throughput. Please see demo videos here - https://aaxwaz.github.io/TalkingMachines/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。