用大模型零样本控制对话规划,避免幻觉并提升任务成功率。
Towards Zero-Shot, Controllable Dialog Planning with LLMs
- 通过领域图搜索与剪枝实现零样本对话规划。
- 模拟测试中显著优于现有最先进方法(p<0.0001)。
- 用户实测提升任务成功率,适合医疗法律等敏感场景。
近年来,大语言模型(LLMs)因其广泛的推理能力和零样本学习表现,成为替代特定任务对话代理的候选方案。然而,多数基于LLM的对话系统在追求整体对话目标的规划能力上表现不足,难以有效引导对话。此外,这些模型易产生幻觉,不适用于法律、医疗等对信息准确性要求极高的敏感领域。最近提出的对话树搜索(Conversational Tree Search, CTS)通过对话图结构来避免幻觉,但当前最先进的代理依赖强化学习(RL),需长时间训练,尽管其对话策略表现优异。本文提出一种新的零样本可控CTS方法,让LLMs通过搜索和剪枝相关图节点,根据用户交互偏好指导对话规划。实验表明,该方法在模拟环境中显著优于现有最先进代理(p<0.0001;Barnard精确检验),且泛化至所有可用的CTS领域。最后,用户评估显示,该策略在真实场景中显著提升任务成功率(p<0.05;Barnard精确检验),优于基于RL的最新代理。
原文摘要 · Abstract (English)
Recently, Large Language Models (LLMs) have emerged as an alternative to training task-specific dialog agents, due to their broad reasoning capabilities and performance in zero-shot learning scenarios. However, many LLM-based dialog systems fall short in planning towards an overarching dialog goal and therefore cannot steer the conversation appropriately. Furthermore, these models struggle with hallucination, making them unsuitable for information access in sensitive domains, such as legal or medical domains, where correctness of information given to users is critical. The recently introduced task Conversational Tree Search (CTS) proposes the use of dialog graphs to avoid hallucination in sensitive domains, however, state-of-the-art agents are Reinforcement Learning (RL) based and require long training times, despite excelling at dialog strategy. This paper introduces a novel zero-shot method for controllable CTS agents, where LLMs guide the dialog planning through domain graphs by searching and pruning relevant graph nodes based on user interaction preferences. We show that these agents significantly outperform state-of-the-art CTS agents ($p<0.0001$; Barnard Exact test) in simulation. This generalizes to all available CTS domains. Finally, we perform user evaluation to test the agent's performance in the wild, showing that our policy significantly ($p<0.05$; Barnard Exact) improves task-success compared to the state-of-the-art RL-based CTS agent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。