基于关键帧的视频生成，让音乐驱动动画更精准流畅。
KeyVID: Keyframe-Aware Video Diffusion for Audio-Synchronized Visual Animation
- 根据音频定位关键帧时间点，动态生成视觉关键帧
- 在低帧率下仍能捕捉剧烈动作的关键时刻，同步性提升明显
- 适合需要高动态、强节奏的音乐视频生成任务
从文本、图像和音频等多种条件生成视频，可实现空间与时间上的双重控制，从而获得高质量结果。具有剧烈运动的视频通常需要更高帧率以保证流畅性。当前多数音视频动画模型采用视频片段均匀采样帧，但在低帧率下难以捕捉剧烈动作中的关键瞬间，直接增加帧数又显著提升内存消耗。本文提出KeyVID，一种面向音视频同步的帧关键帧感知生成框架，在保持计算效率的同时显著提升关键时刻的生成质量。给定一张图像和一段音频，首先从音频中定位关键帧时间步，再通过关键帧生成器生成对应视觉关键帧，最后由运动插值器完成所有中间帧生成。大量实验表明，KeyVID在多个数据集上显著提升了音视频同步性和视频质量，尤其在高度动态动作场景中表现突出。代码已开源：https://github.com/XingruiWang/KeyVID。
原文摘要 · Abstract (English)
Generating video from various conditions, such as text, image, and audio, enables both spatial and temporal control, leading to high-quality generation results. Videos with dramatic motions often require a higher frame rate to ensure smooth motion. Currently, most audio-to-visual animation models use uniformly sampled frames from video clips. However, these uniformly sampled frames fail to capture significant key moments in dramatic motions at low frame rates and require significantly more memory when increasing the number of frames directly. In this paper, we propose KeyVID, a keyframe-aware audio-to-visual animation framework that significantly improves the generation quality for key moments in audio signals while maintaining computation efficiency. Given an image and an audio input, we first localize keyframe time steps from the audio. Then, we use a keyframe generator to generate the corresponding visual keyframes. Finally, we generate all intermediate frames using the motion interpolator. Through extensive experiments, we demonstrate that KeyVID significantly improves audio-video synchronization and video quality across multiple datasets, particularly for highly dynamic motions. The code is released in https://github.com/XingruiWang/KeyVID.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。