arXiv:2604.02908cs.CVcs.HC2026-04被引 3

打造能实时说话、比手势、表情绪的虚拟人,对话自然流畅。

SentiAvatar: Towards Expressive and Interactive Digital Humans

论文配图:SentiAvatar: Towards Expressive and Interactive Digital Humans
图 1 · 摘自论文原文
  • 用分步生成架构解耦语义规划与语音节奏,提升动作自然度。
  • 在21000段数据上达到43.64%准确率,比基线高近一倍。
  • 适合虚拟主播、游戏角色、智能客服等需要情感互动的场景。

我们提出SentiAvatar框架,构建可实时表达与交互的3D虚拟人,并基于此创建了名为SuSu的角色,实现语音、手势与情绪同步。该系统面临三大挑战:缺乏大规模高质量多模态数据、语义到动作映射不鲁棒、帧级动作与语音韵律不同步。为此,我们构建了包含21,000段视频、总时长37小时的SuSuInterActs数据集,涵盖语音、全身动作与面部表情;预训练了一个基于20万+动作序列的运动基础模型,引入丰富动作先验;提出音频感知的“先规划后补全”架构,分离句级语义规划与帧级韵律驱动插值,使生成动作既语义合理又与语音节奏对齐。实验表明,SentiAvatar在SuSuInterActs数据集上达到R@1 43.64%,接近最佳基线两倍;在BEATv2上取得FGD 4.941、BC 8.078,6秒输出仅需0.3秒,支持无限轮次流式生成。代码、模型与数据集已开源。

原文摘要 · Abstract (English)

We present SentiAvatar, a framework for building expressive interactive 3D digital humans, and use it to create SuSu, a virtual character that speaks, gestures, and emotes in real time. Achieving such a system remains challenging, as it requires jointly addressing three key problems: the lack of large-scale, high-quality multimodal data, robust semantic-to-motion mapping, and fine-grained frame-level motion-prosody synchronization. To solve these problems, first, we build SuSuInterActs (21K clips, 37 hours), a dialogue corpus captured via optical motion capture around a single character with synchronized speech, full-body motion, and facial expressions. Second, we pre-train a Motion Foundation Model on 200K+ motion sequences, equipping it with rich action priors that go well beyond the conversation. We then propose an audio-aware plan-then-infill architecture that decouples sentence-level semantic planning from frame-level prosody-driven interpolation, so that generated motions are both semantically appropriate and rhythmically aligned with speech. Experiments show that SentiAvatar achieves state-of-the-art on both SuSuInterActs (R@1 43.64%, nearly 2 times the best baseline) and BEATv2 (FGD 4.941, BC 8.078), producing 6s of output in 0.3s with unlimited multi-turn streaming. The source code, model, and dataset are available at https://sentiavatar.github.io.

虚拟人动作生成语音同步多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。