arXiv:2605.26236cs.CVcs.SD2026-05

分离语义与节奏流,生成更自然的伴随言语手势。

DuoGesture: Neuro-Inspired and Biomechanically Informed Dual-Stream Co-Speech Gesture Generation

论文配图:DuoGesture: Neuro-Inspired and Biomechanically Informed Dual-Stream Co-Speech Gesture Generation
图 1 · 摘自论文原文
  • 双流架构:语义流与节奏流并行,通过随机门控协调。
  • 语义流用动作-语言表征,提升长尾词汇触发的准确性。
  • 引入惯性节律先验,减少抖动,增强节奏一致性。

伴随言语手势生成需兼具语义表达力与生物力学合理性。现有整体模型将语义手势与韵律对齐的节拍手势混合,限制了语义基础、语音-动作对齐及运动平滑性。本文提出神经启发且生物力学引导的双流方法DuoGesture,将手势合成分解为耦合的语义流与节拍流。两流由语义变分信息瓶颈协调,该随机帧级门控学习何时语义手势应覆盖节奏节拍。语义流采用动作-语言条件化,以运动-语言表征替代纯语言词嵌入,为长尾词汇触发提供运动对齐的语义先验。节拍流通过惯性节律先验(anthropometry-weighted arm-chain module)进一步正则化,降低抖动,提升节奏一致性,不约束语义帧。客观评估与主观实验表明,DuoGesture优于强基线;组件消融证实语义锚定、随机流选择与生物力学正则化的互补作用。

原文摘要 · Abstract (English)

Co-speech gesture generation requires both semantic expressivity and biomechanically plausible rhythmic motion. Existing holistic gesture models mix lexically grounded semantic gestures with frequent prosody-aligned beat gestures. This limits semantic grounding, speech-motion alignment, and kinematic smoothness. We propose \emph{DuoGesture}, a neuro-inspired and biomechanically informed dual-stream approach that decomposes co-speech gesture synthesis into coupled semantic and beat streams. The two streams are coordinated by a \emph{Semantic Variational Information Bottleneck}, a stochastic frame-level gate that learns when semantic gestures should override rhythmic beat motion. The semantic stream is controlled by \emph{Motion-Grounded Semantic Conditioning}, which replaces purely linguistic word embeddings with motion-language representations to provide motion-aligned semantic priors for long-tailed lexical triggers of gestures. The beat stream is further regularised by an \emph{Inertial Beat Prior}, an anthropometry-weighted arm-chain module that reduces jitter and improves rhythmic consistency without constraining semantic frames. Objective evaluations and subjective experiments show that DuoGesture outperforms strong holistic baselines, while component ablations confirm the complementary roles of semantic grounding, stochastic stream selection, and biomechanical regularisation.

手势生成双流模型生物力学语音对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。