arXiv:2503.13229cs.CV2025-03中稿 · 3DV 2025被引 9

用解耦扩散与运动先验生成自然连贯的说话手势,提升虚拟角色表现力。

HoloGest: Decoupled Diffusion and Motion Priors for Generating Holisticly Expressive Co-speech Gestures

  • 分离音频与运动先验,利用大规模动作数据学习低依赖、高保真的运动模式。
  • 结合隐式与显式约束,生成速度显著提升,仍保持高质量动作细节。
  • 共享嵌入空间对齐文本与手势,确保语义准确,适合虚拟角色动画应用。

为虚拟角色生成整体协调的说话手势是一项挑战性任务。现有方法多关注语音与手势间微弱关联,导致动作不自然,影响用户体验。为此,我们提出HoloGest,一种基于解耦扩散与运动先验的神经网络框架,实现高质量、富有表现力的自动手势生成。系统利用大规模人体动作数据学习具有低音频依赖性、高动作依赖性的稳健先验,支持稳定的全局运动和精细的手指动作。为提升扩散模型生成效率,我们融合隐式联合约束与显式几何及条件约束,捕捉长距离步幅间的复杂运动分布,显著加速生成过程且保持高质量。此外,设计共享嵌入空间实现手势与文本转录的对齐,确保动作语义正确。大量实验与用户反馈验证了模型的有效性,生成效果接近真实标注数据,提供沉浸式体验。代码、模型与演示见https://cyk990422.github.io/HoloGest.github.io/。

原文摘要 · Abstract (English)

Animating virtual characters with holistic co-speech gestures is a challenging but critical task. Previous systems have primarily focused on the weak correlation between audio and gestures, leading to physically unnatural outcomes that degrade the user experience. To address this problem, we introduce HoleGest, a novel neural network framework based on decoupled diffusion and motion priors for the automatic generation of high-quality, expressive co-speech gestures. Our system leverages large-scale human motion datasets to learn a robust prior with low audio dependency and high motion reliance, enabling stable global motion and detailed finger movements. To improve the generation efficiency of diffusion-based models, we integrate implicit joint constraints with explicit geometric and conditional constraints, capturing complex motion distributions between large strides. This integration significantly enhances generation speed while maintaining high-quality motion. Furthermore, we design a shared embedding space for gesture-transcription text alignment, enabling the generation of semantically correct gesture actions. Extensive experiments and user feedback demonstrate the effectiveness and potential applications of our model, with our method achieving a level of realism close to the ground truth, providing an immersive user experience. Our code, model, and demo are are available at https://cyk990422.github.io/HoloGest.github.io/.

手势生成扩散模型虚拟角色运动先验

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。