实时生成说话手势，用关键姿态锚点防止长期漂移。
StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring
- 每段生成后通过检索数据库关键姿态，闭环修正轨迹。
- 长序列下漂移减少，76帧/秒实时运行，FGD指标领先。
- 适合需要稳定动作输出的虚拟主播、人机交互场景。
实时协同说话手势生成需逐段输出3D动作序列。现有方法为开环：每段依赖历史上下文，但无法验证或修正轨迹，导致小误差累积引发长期漂移。我们发现根本问题在于缺乏前向约束，而非单段质量差。在每段末尾引入合理的关键姿态作为目标锚点，可有效抑制漂移。基于此，提出StreamTalk，采用周期性生成-检索-精炼闭环框架。首先粗略生成一段动作，从说话人专属动作库中检索合理末端姿态，再以此修正该段，随后进入下一段。训练时采用随机锚点掩码（Stochastic Anchor Masking），随机遮蔽姿态与平移帧，训练模型从稀疏边界条件恢复完整运动。部分感知的DiT模型分离手部、身体与平移流，降低全局位移与局部动作间的干扰。在BEAT2数据集上，StreamTalk达到最优FGD指标，显著减少长序列漂移，且实现实时生成，达76 FPS。
原文摘要 · Abstract (English)
Real-time co-speech gesture generation must produce 3D motion clip by clip as speech arrives. Existing streaming methods are open-loop: each clip depends on past context, but the model cannot check or correct its trajectory. Small errors therefore accumulate and cause drift over long sequences. We observe that this failure is mainly caused by the lack of a forward constraint rather than poor short-clip quality. A plausible key pose at the end of each clip provides a destination anchor that limits drift. Based on this observation, we propose StreamTalk, a closed-loop framework with a periodic generate-retrieve-refine cycle. Streaming Pose-Guided Generation first predicts a coarse clip, retrieves a plausible tail pose from a speaker-specific motion database, and refines the clip using this pose before continuing to the next window. During training, Stochastic Anchor Masking randomly masks pose and translation frames, teaching the model to recover complete motion from sparse boundary conditions. A part-aware DiT separates hand, body, and translation streams to reduce interference between global displacement and local articulation. On BEAT2, StreamTalk achieves state-of-the-art FGD, reduces long-horizon drift relative to open-loop baselines, and runs in real time at 76 FPS. Project page: https://xiangyue-zhang.github.io/StreamTalk/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。