用扩散模型生成与音乐同步的逼真舞蹈动作，适合处理社交媒体中的不完整数据。
DanceFusion: A Spatio-Temporal Skeleton Diffusion Transformer for Audio-Driven Dance Motion Reconstruction
- 基于时空骨架的扩散Transformer，融合变分自编码器提升动作质量。
- 在TikTok风格数据上实现高保真舞蹈重建，比现有方法更真实多样。
- 适合内容创作、VR和互动娱乐，可生成动态且风格多样的舞蹈序列。
本文提出DanceFusion，一种用于生成与音乐同步舞蹈动作的新框架，采用时空骨架扩散Transformer。该框架能有效处理社交媒体短视频（如TikTok）中常见的不完整与噪声骨骼数据。DanceFusion结合分层Transformer结构的变分自编码器与扩散模型，显著提升动作的真实感与准确性。通过引入复杂掩码策略和独特的迭代扩散过程，逐步优化运动序列，确保生成动作与音频线索的高度对齐。全面评估表明，DanceFusion在动态性、真实性和风格多样性方面均优于现有方法，达到当前最佳性能。该框架在内容创作、虚拟现实及互动娱乐等领域具有广泛应用潜力。
原文摘要 · Abstract (English)
This paper introduces DanceFusion, a novel framework for reconstructing and generating dance movements synchronized to music, utilizing a Spatio-Temporal Skeleton Diffusion Transformer. The framework adeptly handles incomplete and noisy skeletal data common in short-form dance videos on social media platforms like TikTok. DanceFusion incorporates a hierarchical Transformer-based Variational Autoencoder (VAE) integrated with a diffusion model, significantly enhancing motion realism and accuracy. Our approach introduces sophisticated masking techniques and a unique iterative diffusion process that refines the motion sequences, ensuring high fidelity in both motion generation and synchronization with accompanying audio cues. Comprehensive evaluations demonstrate that DanceFusion surpasses existing methods, providing state-of-the-art performance in generating dynamic, realistic, and stylistically diverse dance motions. Potential applications of this framework extend to content creation, virtual reality, and interactive entertainment, promising substantial advancements in automated dance generation. Visit our project page at https://th-mlab.github.io/DanceFusion/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。