用自然对话训练无需标注的对话系统,提升跨领域任务完成率。
Evaluating and Enhancing Out-of-Domain Generalization of Task-Oriented Dialog Systems for Task Completion without Turn-level Dialog Annotations
- 仅用自然对话微调大模型,避开繁琐标注
- 无标注模型在未知领域任务成功率仅53%
- 提出ZeroToD框架,显著提升跨域任务完成度
传统任务导向对话系统依赖人工标注的对话状态和策略标签进行训练。本文探索仅用自然语言对话微调大语言模型(LLMs)完成任务导向对话任务的可行性,无需转轮级标注。在三个不同规模的开源LLM与两个多样化任务导向对话数据集上进行实验,发现无标注微调模型能生成连贯且符合上下文的回复,但任务完成率(以准确执行API调用衡量)仍不理想,最优模型在未知领域仅达约53%成功率。为此,本文提出ZeroToD框架,引入模式增强机制,显著提升API调用准确率与整体任务完成率,尤其在跨域场景下表现突出。对比零微调方案(如直接提示),所提框架使小型微调模型优于大型专有LLM。人工评估也验证了其在信息量、流畅性与任务完成上的优势。结果表明,低成本、可扩展、零样本泛化的任务导向对话系统具有现实应用前景。
原文摘要 · Abstract (English)
Traditional task-oriented dialog (ToD) systems rely heavily on labor-intensive turn-level annotations, such as dialogue states and policy labels, for training. This work explores whether large language models (LLMs) can be fine-tuned solely on natural language dialogs to perform ToD tasks, without requiring such annotations. We evaluate their ability to generalize to unseen domains and compare their performance with models trained on fully annotated data. Through extensive experiments with three open-source LLMs of varying sizes and two diverse ToD datasets, we find that models fine-tuned without turn-level annotations generate coherent and contextually appropriate responses. However, their task completion performance - measured by accurate execution of API calls - remains suboptimal, with the best models achieving only around 53% success in unseen domains. To improve task completion, we propose ZeroToD, a framework that incorporates a schema augmentation mechanism to enhance API call accuracy and overall task completion rates, particularly in out-of-domain settings. We also compare ZeroToD with fine-tuning-free alternatives, such as prompting off-the-shelf LLMs, and find that our framework enables smaller, fine-tuned models that outperform large-scale proprietary LLMs in task completion. Additionally, a human study evaluating informativeness, fluency, and task completion confirms our empirical findings. These findings suggest the feasibility of developing cost-effective, scalable, and zero-shot generalizable ToD systems for real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。