用大模型驱动双人对话中的自然非语言行为生成
Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agents
- 基于大模型构建对话代理，实时决策双方互动行为
- 提出自回归扩散模型同步生成双人手势动作
- 通过反馈循环实现动态响应，提升交互真实感
我们提出 Social Agent，一个用于合成双人对话中真实且符合语境的伴随性非语言行为的新框架。该框架采用由大型语言模型（LLM）驱动的代理系统，引导对话流程并决定双方的互动行为。同时，我们设计了一种基于自回归扩散模型的双人手势生成模型，从语音信号中合成协调的动作。代理系统的输出转化为手势生成器的高层指导，从而在行为与运动层面生成逼真的动作。此外，代理系统定期分析对话者动作并推断其意图，形成持续的反馈回路，实现双参与者间的动态响应。用户研究与定量评估表明，该模型显著提升了双人交互质量，生成了自然、同步的非语言行为。
原文摘要 · Abstract (English)
We present Social Agent, a novel framework for synthesizing realistic and contextually appropriate co-speech nonverbal behaviors in dyadic conversations. In this framework, we develop an agentic system driven by a Large Language Model (LLM) to direct the conversation flow and determine appropriate interactive behaviors for both participants. Additionally, we propose a novel dual-person gesture generation model based on an auto-regressive diffusion model, which synthesizes coordinated motions from speech signals. The output of the agentic system is translated into high-level guidance for the gesture generator, resulting in realistic movement at both the behavioral and motion levels. Furthermore, the agentic system periodically examines the movements of interlocutors and infers their intentions, forming a continuous feedback loop that enables dynamic and responsive interactions between the two participants. User studies and quantitative evaluations show that our model significantly improves the quality of dyadic interactions, producing natural, synchronized nonverbal behaviors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。