诊断合成对话生成的缺陷,提升真实客服对话生成质量
Why Synthetic Isn't Real Yet: A Diagnostic Framework for Contact Center Dialogue Generation
- 用结构化监督指导多语言对话生成,基于意图、话题流和质检表
- 合成对话在自动质检任务中表现差于真实对话,尤其在情感与互动真实度上
- 提出17项指标框架,覆盖情绪弧、语言复杂度等四维度诊断缺陷
合成数据在客服中心日益重要,因隐私限制和真实对话稀缺。然而,生成真实且可用的合成对话仍具挑战。本文在多语言环境下,基于意图摘要、话题流和质检表等结构化监督,评估多种生成策略。通过自动质检(AutoQA)任务测试下游效用,发现以真实对话优化的提示始终优于以合成对话优化的提示,表明当前合成对话未能充分捕捉真实人机交互的复杂性。为此,我们提出包含17项指标的诊断评估框架,涵盖四个维度:(1) 情绪与情感弧线,(2) 语言复杂度,(3) 互动风格,(4) 对话属性。分析显示,即使有结构化监督,现有方法在情感保真度、不流畅建模、行为多样性及对话真实性方面仍存在明显不足。结果强调了对合成对话生成进行诊断性、指标驱动评估的重要性。
原文摘要 · Abstract (English)
Synthetic data is increasingly critical for contact centers, where privacy constraints and data scarcity limit the availability of real conversations. However, generating synthetic dialogues that are realistic and useful for downstream applications remains challenging. In this work, we benchmark multiple generation strategies guided by structured supervision on call attributes (Intent Summaries, Topic Flows, and Quality Assurance (QA) Forms) across multiple languages. To test downstream utility, we evaluate synthetic transcripts on an automated quality assurance (AutoQA) task, finding that prompts optimized on real transcripts consistently outperform those optimized on synthetic transcripts. These results suggest that current synthetic transcripts fall short in capturing the full realism of real agent-customer interactions. To highlight these downstream gaps, we introduce a diagnostic evaluation framework comprising 17 metrics across four dimensions: (1) Emotional and Sentiment Arcs, (2) Linguistic Complexity, (3) Interaction Style, and (4) Conversational Properties. Our analysis shows that even with structured supervision, current generation strategies exhibit measurable deficiencies in sentiment fidelity, disfluency modeling, behavioral variation, and conversational realism. Together, these results highlight the importance of diagnostic, metric-driven evaluation for synthetic conversation generation intended for downstream applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。