arXiv:2605.06064cs.CV2026-05被引 9

仅用一段视频就能让虚拟人模仿陌生人的说话手势,且不需重新训练。

PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakers

论文配图:PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakers
图 1 · 摘自论文原文
  • 用扩散模型分离说话者风格与语句动作,避免风格混淆。
  • 在两个数据集上比基线方法更准确保留说话者特征,人类偏好更高。
  • 适合做虚拟主播、数字人,只需一个参考视频即可定制动作。

我们提出PersonaGesture,一种基于扩散模型的单参考共言手势个性化方法,适用于未见过的说话人。给定目标语音和一位新说话人的一个动作片段,模型需生成符合该语音的新动作,同时保留说话人特有的姿态习惯,无需针对每个说话人进行微调。该设定对虚拟角色和数字人应用很有价值,但难度在于参考片段中混合了稳定的个人习惯与语句相关的动作轨迹。PersonaGesture包含两个关键组件:自适应风格注入(ASI)和隐式分布修正(IDR),用于分离时间维度的身份特征与残差统计修正。风格感知器首先将可变长度的参考序列编码为紧凑的说话人记忆标记。ASI通过零初始化的残差交叉注意力将这些标记注入去噪过程,使风格信息影响动作生成,而不替换预训练的语音到动作先验。在此基础上,IDR在潜在空间中使用长度感知的对角仿射变换,修正从同一参考中估计出的通道级残差均值与方差。在BEAT2和ZeroEGGS数据集上,我们评估了定量指标、参考身份控制、同音频诊断、定性对比和人类偏好。实验表明,将去噪阶段的说话人记忆与保守的生成后分布修正相分离,相比坍缩风格码、全参考注意力和单片段微调,在未见说话人个性化任务上表现更优。

原文摘要 · Abstract (English)

We propose PersonaGesture, a diffusion-based pipeline for single-reference co-speech gesture personalization of unseen speakers. Given target speech and one motion clip from a new speaker, the model must synthesize gestures that follow the new utterance while retaining speaker-specific pose choices, without per-speaker optimization. This setting is useful for avatars and virtual agents, but it is hard because the reference mixes stable speaker habits with utterance-specific trajectories. PersonaGesture consists of two key components, Adaptive Style Infusion (ASI) and Implicit Distribution Rectification (IDR), to separate temporal identity evidence from residual statistic correction. A Style Perceiver first encodes the variable-length reference into compact speaker-memory tokens. ASI injects these tokens into denoising through zero-initialized residual cross-attention, enabling style evidence to affect motion formation without replacing the pretrained speech-to-motion prior. Building on this, IDR applies a length-aware diagonal affine map in latent space to correct residual channel-wise moments estimated from the same reference. Across BEAT2 and ZeroEGGS, we evaluate quantitative metrics, reference-identity controls, same-audio diagnostics, qualitative comparisons, and human preference. Experiments show that separating denoising-time speaker memory from conservative post-generation moment correction improves unseen-speaker personalization over collapsed style codes, full-reference attention, and one-clip finetuning. Project: https://xiangyue-zhang.github.io/PersonaGesture.

手势生成个性定制扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。