通过问责机制减少用户对对话AI的过度依赖,提升错误检测与纠正能力。
Know Your Mistakes: Towards Preventing Overreliance on Task-Oriented Conversational AI Through Accountability Modeling
- 在大模型中加入问责头,判断对话状态中的关键信息是否准确。
- 使对话系统联合目标准确率提升3%,最高达70.51%。
- 通过用户确认环节引入摩擦,有效降低用户盲信错误建议的风险。
近年来的大语言模型显著推动了对话代理的发展,但其也普遍存在幻觉问题,生成看似合理却事实错误的回答。用户常过度信任这些AI,即使出错也接受建议。本文提出一种针对任务导向对话系统的问责模型,通过在模型不确定或对话状态跟踪(DST)出错时引入摩擦回合(如解释或用户确认),缓解用户过度依赖问题。该模型在多个骨干LLM上测试,使用MultiWOZ和Snips两个基准数据集。实验表明,引入问责头可实现更可靠的错误估计,并引导解码器生成更准确动作,使联合目标准确率(JGA)绝对提升约3%。通过自我纠错,JGA从67.13提高至70.51,达到当前最优水平。进一步验证用户确认带来的纠错机制能实现相似性能提升,证明其在减少用户过信方面具有潜力。
原文摘要 · Abstract (English)
Recent LLMs have enabled significant advancements for conversational agents. However, they are also well known to hallucinate, producing responses that seem plausible but are factually incorrect. On the other hand, users tend to over-rely on LLM-based AI agents, accepting AI's suggestion even when it is wrong. Adding positive friction, such as explanations or getting user confirmations, has been proposed as a mitigation in AI-supported decision-making systems. In this paper, we propose an accountability model for LLM-based task-oriented dialogue agents to address user overreliance via friction turns in cases of model uncertainty and errors associated with dialogue state tracking (DST). The accountability model is an augmented LLM with an additional accountability head that functions as a binary classifier to predict the relevant slots of the dialogue state mentioned in the conversation. We perform our experiments with multiple backbone LLMs on two established benchmarks (MultiWOZ and Snips). Our empirical findings demonstrate that the proposed approach not only enables reliable estimation of AI agent errors but also guides the decoder in generating more accurate actions. We observe around 3% absolute improvement in joint goal accuracy (JGA) of DST output by incorporating accountability heads into modern LLMs. Self-correcting the detected errors further increases the JGA from 67.13 to 70.51, achieving state-of-the-art DST performance. Finally, we show that error correction through user confirmations (friction turn) achieves a similar performance gain, highlighting its potential to reduce user overreliance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。