分离对话轮次与语义,用真实对话数据提升全双工对话自然度。
Decoupling Turn-Taking from Semantics: A Decoupled Data Approach for Finite-State-Machine-Based Full-Duplex Dialogue

- 用真实人类对话数据学习轮次控制,文本对话数据保留语义能力。
- 通过事件分类和规则映射,将真人对话转为状态机可处理的序列。
- 自适应损失函数让不同数据源各司其职,显著提升轮次判断准确率。
神经有限状态机(NFSM)框架通过将轮次控制与响应生成统一在单个因果序列上,以标准的下一个词预测目标实现全双工对话,同时保持语义能力且微调成本低。然而,其依赖合成文本数据的根本缺陷在于,大语言模型无法真实模拟人类对话中精细的声学时间动态,导致轮次交互不自然。为此,本文提出解耦数据方法:从真实人-人(HH)语音对话中学习轮次控制,通过可配置的人-代理(HA)文本对话塑造语义行为。我们设计基于规则的事件引导数据转换方法,通过分类轮次事件并应用确定性映射规则,将HH语音对话序列化为状态机可处理的序列,实现无需LLM标注的大规模监督。进一步提出源感知校准(SAC)损失,联合校准状态转移标记的长尾分布,并引导每类数据源聚焦于其最优监督能力。实验表明,该方法显著提升轮次判断能力,同时完全恢复基础大模型的语义性能。代码与模型已开源:https://github.com/Liyht/def-fsm。
原文摘要 · Abstract (English)
The Neural Finite State Machine (NFSM) framework offers a pragmatic path to full-duplex dialogue by serializing turn-taking control and response generation onto a single causal tape under the standard next-token prediction objective, thereby preserving semantic prowess at a low fine-tuning cost. However, its reliance on synthetic text data fundamentally limits turn-taking naturalness, as Large Language Models (LLMs) cannot faithfully simulate the fine-grained acoustic temporal dynamics of real human dialogues. In this work, we propose a decoupled data approach that learns turn-taking from real Human-Human (HH) spoken dialogues while shaping semantic behavior through configurable Human-Agent (HA) text dialogues. To operationalize this approach, we introduce a rule-based event-guided data transformation method that serializes HH spoken dialogues into FSM tapes by classifying turn-taking events and applying deterministic mapping rules, enabling scalable supervision without LLM-generated annotations. We further propose a Source-Aware Calibrated (SAC) Loss that jointly calibrates the long-tailed distribution of state transition tokens and channels each data source toward the capability it best supervises. Experiments show that our approach substantially improves turn-taking proficiency while recovering the foundation LLM's semantic capability. Our code and model are available at https://github.com/Liyht/def-fsm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。