arXiv:2509.10010cs.CLcs.HC2025-09被引 2

比较三款小模型在对话多意图识别中的表现,发现小模型也能高效处理复杂对话。

Multi-Intent Recognition in Dialogue Understanding: A Comparison Between Smaller Open-Source LLMs

  • 用20个示例提示,测试三款开源小模型在少样本下的多意图分类能力。
  • Mistral-7B在14个意图中11个表现最佳,加权F1达0.50,推理更快显存更低。
  • 基于BERT的监督模型反而优于最优生成式模型,适合追求精度的开发者。

本文针对可在消费级硬件运行的开源大语言模型(LLMs),在多标签意图分类任务中开展全面分析。使用对话系统领域的基准数据集MultiWOZ 2.1,评估LLama2-7B-hf、Mistral-7B-v0.1和Yi-6B三款预训练模型在少样本设置下的表现,每个任务提供20个示例及指令。通过准确率、精确率、召回率以及微平均、宏平均和加权F1等指标评估性能,并记录推理时间与显存占用。结果显示,Mistral-7B-v0.1在14个意图类别中的11个上优于其他两个生成式模型,加权平均F1为0.50,同时具有较低的汉明损失和较高的杰卡德相似度,是少样本场景下的优解。相比之下,基于BERT的监督学习模型表现更佳,优于当前最优的少样本生成式模型。研究为小型开源模型在复杂多意图对话理解中的应用提供了可复现框架,有助于提升任务导向聊天机器人的自然语言理解能力。

原文摘要 · Abstract (English)

In this paper, we provide an extensive analysis of multi-label intent classification using Large Language Models (LLMs) that are open-source, publicly available, and can be run in consumer hardware. We use the MultiWOZ 2.1 dataset, a benchmark in the dialogue system domain, to investigate the efficacy of three popular open-source pre-trained LLMs, namely LLama2-7B-hf, Mistral-7B-v0.1, and Yi-6B. We perform the classification task in a few-shot setup, giving 20 examples in the prompt with some instructions. Our approach focuses on the differences in performance of these models across several performance metrics by methodically assessing these models on multi-label intent classification tasks. Additionally, we compare the performance of the instruction-based fine-tuning approach with supervised learning using the smaller transformer model BertForSequenceClassification as a baseline. To evaluate the performance of the models, we use evaluation metrics like accuracy, precision, and recall as well as micro, macro, and weighted F1 score. We also report the inference time, VRAM requirements, etc. The Mistral-7B-v0.1 outperforms two other generative models on 11 intent classes out of 14 in terms of F-Score, with a weighted average of 0.50. It also has relatively lower Humming Loss and higher Jaccard Similarity, making it the winning model in the few-shot setting. We find BERT based supervised classifier having superior performance compared to the best performing few-shot generative LLM. The study provides a framework for small open-source LLMs in detecting complex multi-intent dialogues, enhancing the Natural Language Understanding aspect of task-oriented chatbots.

多意图识别小模型对话系统少样本学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。