arXiv:2409.18568cs.AIcs.LG2024-09被引 8

对比多种模型在客服对话系统中的表现,找出最优组合。

Experimental Evaluation of Machine Learning Models for Goal-oriented Customer Service Chatbot with Pipeline Architecture

  • 分模块评估NLU、DM、NLG各组件的ML模型性能。
  • BERT适合意图识别,LSTM在槽位填充上更优;DDQN比DQN更高效。
  • 大模型GPT-2在生成质量上全面超越DialoGPT,适合实用部署。

将机器学习(ML)引入客服聊天机器人可提升其理解与应答能力,从而改善服务表现。然而,部分用户可能感觉其回应生硬,影响体验。因此,对流水线架构中各组件的ML模型进行细致评估至关重要,但功能差异可能导致比较不公平。本文提出一种针对目标导向型客服聊天机器人的定制化实验评估方法,聚焦自然语言理解(NLU)、对话管理(DM)和自然语言生成(NLG)三个核心组件。方法强调独立评估以确定最优模型。具体地,优化超参数并评估候选模型:NLU采用BERT与LSTM,DM使用DQN与DDQN,NLG则对比GPT-2与DialoGPT。结果表明,在NLU中,BERT在意图识别上表现更优,而LSTM在槽位填充上更佳;在DM中,DDQN模型在更少轮次内达成更高奖励与成功率;在NLG方面,GPT-2在BLEU、METEOR与ROUGE指标上均优于DialoGPT。这些发现为未来客服机器人研发与优化提供基准参考,揭示了各模型性能及最优超参数配置。

原文摘要 · Abstract (English)

Integrating machine learning (ML) into customer service chatbots enhances their ability to understand and respond to user queries, ultimately improving service performance. However, they may appear artificial to some users and affecting customer experience. Hence, meticulous evaluation of ML models for each pipeline component is crucial for optimizing performance, though differences in functionalities can lead to unfair comparisons. In this paper, we present a tailored experimental evaluation approach for goal-oriented customer service chatbots with pipeline architecture, focusing on three key components: Natural Language Understanding (NLU), dialogue management (DM), and Natural Language Generation (NLG). Our methodology emphasizes individual assessment to determine optimal ML models. Specifically, we focus on optimizing hyperparameters and evaluating candidate models for NLU (utilizing BERT and LSTM), DM (employing DQN and DDQN), and NLG (leveraging GPT-2 and DialoGPT). The results show that for the NLU component, BERT excelled in intent detection whereas LSTM was superior for slot filling. For the DM component, the DDQN model outperformed DQN by achieving fewer turns, higher rewards, as well as greater success rates. For NLG, the large language model GPT-2 surpassed DialoGPT in BLEU, METEOR, and ROUGE metrics. These findings aim to provide a benchmark for future research in developing and optimizing customer service chatbots, offering valuable insights into model performance and optimal hyperparameters.

客服机器人对话系统模型评估NLU

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。