让虚拟人说话时同步生成全身动作,一次推理完成,速度更快、更自然。
Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

- 端到端联合生成语音与面部、手部、躯干和腿部动作,直接从语音隐状态输出
- 在1402小时数据上训练,动作与语音对齐度接近教师模型,响应速度提升5.4倍
- 首次公开开放对话的全身体态评估协议,适合虚拟人交互、数字主播等场景
一个能对话的虚拟人需同时决定说什么以及如何动,但现有模型分属不同体系:对话模型生成语音无动作,动作模型仅根据音频生成动作。传统方法是级联:先生成语音,再用动作模型处理音频,需二次推理且无法联合优化。本文提出Motion-Omni,一种端到端框架,使对话模型原生输出面部表情及手、上身、下身动作,直接由生成语音的隐状态驱动。联合训练不可或缺:冻结语音路径后,动作仍失准;唯有同时优化大语言模型、语音生成器与动作生成器,才能恢复对齐并保留对话能力。监督来自可扩展、模型无关的伪标签流水线,以可替换的动作教师为1,402小时高质量语音-动作对(共422,856组)打标。我们进一步发布SwDA-500,并首次公开开放式全身体态对话的评估协议,统一渲染、自动指标、人工评价与延迟测量。基于Qwen2.5-7B-Instruct,Motion-Omni-Q7在无参考动作指标上比同音频教师级联低2%,实时因子(RTF)达0.78(超实时),且在节拍相关性与多样性上超越所有非教师级联方案,词错误率仅2.62%,为当前最先进多模态系统最低。
原文摘要 · Abstract (English)
An avatar that holds a conversation should decide what to say and to move while saying it, yet these abilities live in separate model families: spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them. The standard remedy is a cascade that first generates the spoken response and then runs a motion model over the finished audio, which requires a second full inference pass and precludes any joint optimisation between the two. We present Motion-Omni, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression together with hand, upper-body and lower-body motion, generated directly from the hidden states that produce the speech. Joint training is not optional here: with the speech pathway frozen, motion remains misaligned with the audio, and co-adapting the LLM, Speech Generator and Motion Generator under both objectives is what recovers alignment while retaining spoken-dialogue ability. Supervision comes from a scalable, model-agnostic pipeline that pseudo-labels consistent-voice speech responses with a replaceable motion teacher, yielding 422,856 quality-ranked pairs (1,402 hours). We further release SwDA-500 and, to our knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems while unifying rendering, automatic metrics, human evaluation, and latency measurement. Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding 5.4 x faster (RTF=0.78, faster than real time), surpasses all non-teacher cascades on beat correlation and diversity, and reaches a 2.62% word error rate, the lowest among the omni-modal systems compared.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。