用离线强化学习提升多领域对话系统任务完成率
Improving Multi-Domain Task-Oriented Dialogue System with Offline Reinforcement Learning
- 基于GPT2的统一模型，结合监督与强化学习优化
- 在MultiWOZ2.1上提升信息告知率1.60%、成功率3.17%
- 适合关注任务导向对话性能提升的研究者
任务导向对话（TOD）系统旨在通过对话完成用户定义的任务。随着预训练大语言模型的发展，TOD系统逐渐采用端到端建模。仅使用监督学习微调会导致暴露偏差和标记丢失问题，使模型偏离用户任务目标。为此，本文提出一种基于统一预训练语言模型GPT2的TOD系统，结合监督学习与强化学习进行优化。采用非可微奖励函数，其计算基于成功概率与BLEU指标的加权和，引导模型同时实现任务完成与语句流畅性。模型在对话会话层级（包含用户话语、信念状态、系统动作和系统回复）上进行微调。在MultiWOZ2.1数据集上的实验表明，相比基线模型，本方法将信息告知率提升1.60%，成功率提升3.17%。
原文摘要 · Abstract (English)
Task-oriented dialogue (TOD) system is designed to accomplish user-defined tasks through dialogues. The TOD system has progressed towards end-to-end modeling by leveraging pre-trained large language models. Fine-tuning the pre-trained language models using only supervised learning leads to the exposure bias and token loss problem and it deviates the models from completing the user's task. To address these issues, we propose a TOD system that leverages a unified pre-trained language model, GPT2, as a base model. It is optimized using supervised learning and reinforcement learning (RL). The issues in the TOD system are mitigated using a non-differentiable reward function. The reward is calculated using the weighted sum of the success rate and BLEU evaluation metrics. The success rate and BLEU metrics in reward calculation guide the language model for user task completion while ensuring a coherent and fluent response. Our model is acquired by fine-tuning a pre-trained model on the dialogue-session level which comprises user utterance, belief state, system act, and system response. Experimental results on MultiWOZ2.1 demonstrate that our model increases the inform rate by 1.60% and the success rate by 3.17% compared to the baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。