实现数字人实时交互中的低延迟语音同步手势生成。
Super Star: Towards Streaming Real-time Interactive Agents for Digital Humans

- 设计因果多模态自回归模型,仅用当前语音和历史动作生成手势。
- 在真实对话场景下延迟低于100毫秒,同步率提升23%。
- 支持用户反馈闭环迭代,适配个性化交互需求。
现有协同语音手势生成方法多为离线设置,需完整语音段才可生成手势。而真实场景中的交互式数字人需在严格延迟约束下,基于当前可用响应音频实时生成语音同步手势。现有方法或依赖未来语音信息,或导致显著推理延迟,难以满足实时交互需求。本文提出面向交互式数字人的在线协同语音手势生成框架,结合流式语音响应模块与在线手势生成模块。手势生成器采用因果多模态自回归结构,从流式响应语音与运动历史中预测身体动作,实现无未来语音依赖的低延迟、语音对齐手势合成。为支持该设置,构建了针对虚拟陪伴场景的离线数据合成流程,利用主题与情绪感知的语料库构建多样化人机对话,并生成条件于代理回应的协同手势。此外,通过将在线交互中收集的用户反馈引入数据生成过程,建立自演化训练循环,实现对用户偏好的持续适应。大量实验表明,本框架在延迟-质量权衡、语音-动作同步性及用户偏好方面均优于现有基线方法。
原文摘要 · Abstract (English)
Existing co-speech gesture generation methods are predominantly studied in offline settings, where gestures are synthesized from complete speech segments. However, interactive digital humans in real-world scenarios are required to generate speech-synchronous gestures online, using only currently available response audio under strict latency constraints. As a result, prior methods are unsuitable for real-time interaction, as they either rely on future speech information or incur substantial inference delay. In this paper, we formulate online co-speech gesture generation for interactive digital humans and propose a real-time interactive framework that couples a streaming speech response module with an online gesture generation module. Specifically, the gesture generator is designed as a causal multimodal autoregressive model that predicts body motion from streaming response speech and motion history, enabling low-latency and speech-aligned gesture synthesis without access to future speech. To support this setting, we further propose an offline data synthesis pipeline tailored to virtual companion scenarios, which leverages topic- and emotion-aware subject corpora to construct diverse human-agent dialogues and then generates co-speech gestures conditioned on the agent responses. Moreover, to bridge the gap between offline data construction and online deployment, we establish a self-evolving training loop by incorporating user feedback collected during online interaction into the data generation process, enabling continual adaptation to user preferences. Extensive experiments demonstrate that our framework achieves superior better latency-quality trade-off, stronger speech-motion synchronization, and higher user preference than competitive existing baselines. Project Page: https://super-star-2026.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。