让语音对话模型同步生成人脸表情，实现自然的双工交互。
Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems
- 用编码器将人脸视频转为离散表情符号，支持快速重建。
- 新增非自回归人脸生成模块，实现音视频实时同步。
- 保持原有语音质量的同时，提升对话的视觉自然度。
全双工语音对话模型（如 Moshi）可实现低延迟自然对话，但仅限于音频模态，缺乏人类交流中不可或缺的表情表达。本文提出 Moshi-Face，首个同时处理用户语音与面部输入，并同步生成语音和面部动作的全双工对话模型。首先构建一个向量量化变分自编码器（VQ-VAE）作为人脸编解码器，将从面部视频提取的3D头像网格编码为紧凑的离散令牌（称为面令牌），并可逆重构3D网格。随后在 Moshi 中引入面令牌转换器模块，以非自回归方式生成面令牌，使 Moshi-Face 能够实时生成同步的音频与面令牌。实验表明，Moshi-Face 在低延迟下实现了音视频对齐，同时保留了原始音频模型的对话质量。
原文摘要 · Abstract (English)
Full-duplex spoken dialogue models, such as Moshi, enable natural, low-latency voice conversations. However, they remain limited to the audio modality, lacking the facial expressions that are integral to human communication. We present Moshi-Face, the first full-duplex dialogue model that jointly processes the user's audio and facial input while simultaneously generating speech and facial motion. We first construct a vector-quantized variational autoencoder (VQ-VAE) as a face codec that encodes 3D head meshes extracted from facial videos into compact discrete tokens, referred to as face tokens, and conversely reconstructs 3D meshes from these tokens. We then extend Moshi with a Face Transformer module that generates face tokens non-autoregressively, enabling Moshi-Face to produce synchronized audio and face tokens in real time. Experiments show that Moshi-Face achieves audiovisual alignment at low latency while preserving the dialogue quality of the original audio-only model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。