用大模型辅助聚类客户意图,提升客服对话分析准确率
Dial-In LLM: Human-Aligned LLM-in-the-loop Intent Clustering for Customer Service Dialogues
- 将大模型融入聚类流程,动态优化意图分组和数量
- 中文数据集超10万通通话,1507个标注意图,准确率达95%以上
- 适合需要高精度意图识别的客服自动化系统
发现客户意图对自动化服务代理至关重要,但现有聚类方法常依赖嵌入距离度量,忽视语义结构。为此,本文提出一种大模型在环(LLM-in-the-loop, LLM-ITL)意图聚类框架,将大模型的语言理解能力融入传统聚类算法。具体而言:(1) 验证微调大模型在语义一致性评估与意图命名中的有效性,准确率超过95%,与人工判断高度一致;(2) 设计迭代式LLM-ITL框架,实现连贯意图簇的自动发现及最优簇数确定;(3) 引入上下文感知技术,适配客服对话场景。由于现有英文基准缺乏足够的语义多样性与意图覆盖,我们进一步构建了一个包含超过10万条真实客服通话、1507个人工标注簇的综合性中文意图数据集。所提方法显著优于基于大模型引导的基线,在聚类质量、成本效率及下游应用中均有明显提升。结合多项最佳实践,研究凸显了大模型在环技术在可扩展对话数据挖掘中的优势。
原文摘要 · Abstract (English)
Discovering customer intentions is crucial for automated service agents, yet existing intent clustering methods often fall short due to their reliance on embedding distance metrics and neglect of underlying semantic structures. To address these limitations, we propose an LLM-in-the-loop (LLM-ITL) intent clustering framework, integrating the language understanding capabilities of LLMs into conventional clustering algorithms. Specifically, this paper (1) examines the effectiveness of fine-tuned LLMs in semantic coherence evaluation and intent cluster naming, achieving over 95% accuracy aligned with human judgments; (2) designs an LLM-ITL framework that facilitates the iterative discovery of coherent intent clusters and the optimal number of clusters; and (3) introduces context-aware techniques tailored for customer service dialogue. Since existing English benchmarks lack sufficient semantic diversity and intent coverage, we further present a comprehensive Chinese dialogue intent dataset comprising over 100k real customer service calls with 1,507 human-annotated clusters. The proposed approaches significantly outperform LLM-guided baselines, achieving notable improvements in clustering quality, cost efficiency, and downstream applications. Combined with several best practices, our findings highlight the prominence of LLM-in-the-loop techniques for scalable dialogue data mining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。