让聊天机器人学会何时说话，提升对话自然度
Beyond Words: Multimodal LLM Knows When to Speak
- 用视频音频文本同步数据训练模型，判断该沉默、短答还是长答
- 在真实对话数据上实现响应类型预测性能提升3倍
- 适合做智能助手、虚拟人等需要自然交互的场景
基于大语言模型的聊天机器人虽能生成流畅回复，但在对话中常难以把握发言时机，尤其对短暂、及时的听者反应。本文提出一种多模态策略，利用同步的视频、音频和文本线索提升对话时机感知能力。该策略将回应时机建模为密集响应类型预测任务，使智能体能在流式约束下判断是否保持沉默、发出简短反应或开启完整回应。为此，我们构建了一个来自真实双人对话视频的多模态数据集，包含时间对齐的多模态信息和细粒度反应类型标注。进一步设计了基于LLM主干的多模态集成模块，名为MM-When2Speak。在多种模态设置与强基线对比实验中，该模型在响应类型预测任务上最高提升3倍性能，凸显多模态感知对自然、生动对话交互的关键作用。
原文摘要 · Abstract (English)
Chatbots via large language models (LLMs) generate fluent responses but often struggle with when to speak, especially for brief, timely listener reactions during ongoing dialogue. We present a multimodal strategy for LLMs, which leverages synchronized video, audio, and text cues to improve conversational timing awareness. The strategy reformulates response timing as a dense response-type prediction task, enabling an agent to decide whether to remain silent, produce a short reaction, or start a full response under streaming constraints. Therefore, we introduce a curated multimodal dataset from real-world dyadic conversational videos with temporally aligned modalities and fine-grained reaction type annotations. Moreover, we design a multimodal strategy, MM-When2Speak, with a multimodal integration module on top of an LLM backbone. Experiments across various modality settings and strong LLM baselines show that MM-When2Speak achieves up to a 3x improvement in response type prediction performance, highlighting the importance of multimodal perception for natural and engaging conversational interaction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。