arXiv:2503.13975cs.CLcs.HC2025-03ACL被引 34

研究人与大模型对话中的理解偏差,发现模型很少主动澄清问题。

Navigating Rifts in Human-LLM Grounding: Study and Benchmark

  • 构建对话接地行为分类体系,分析三组真实交互数据
  • 模型发起澄清的概率仅为人类的1/3,跟进请求概率为1/16
  • 提出Rifts基准测试,适合研究对话鲁棒性与人机协作

语言模型在遵循指令方面表现优异,但在人类自然对话中常见的协作性理解(即对话接地)方面存在明显不足。这种接地缺陷可能导致用户挫败感,甚至在高风险场景中引发严重后果。为系统研究人-大模型交互中的接地挑战,我们分析了三个真实人机助手数据集(WildChat、MultiWOZ、Bing Chat)的日志,建立接地行为分类体系,并构建模型用于标注与预测接地行为。研究发现,大模型发起澄清的概率仅为人类的三分之一,提供后续追问的概率更是低至人类的十六分之一。此外,早期接地失败可有效预测后续交互崩溃。基于这些发现,我们推出Rifts基准,该基准源自公开的大模型交互数据,包含大量模型未主动启动接地的情境。结果显示,当前前沿模型在Rifts上的表现不佳,凸显了重新设计训练与提示策略的必要性。为此,我们提出初步干预方案以缓解接地失败问题。

原文摘要 · Abstract (English)

Language models excel at following instructions but often struggle with the collaborative aspects of conversation that humans naturally employ. This limitation in grounding -- the process by which conversation participants establish mutual understanding -- can lead to outcomes ranging from frustrated users to serious consequences in high-stakes scenarios. To systematically study grounding challenges in human-LLM interactions, we analyze logs from three human-assistant datasets: WildChat, MultiWOZ, and Bing Chat. We develop a taxonomy of grounding acts and build models to annotate and forecast grounding behavior. Our findings reveal significant differences in human-human and human-LLM grounding: LLMs were three times less likely to initiate clarification and sixteen times less likely to provide follow-up requests than humans. Additionally, we find that early grounding failures predict later interaction breakdowns. Building on these insights, we introduce Rifts, a benchmark derived from publicly available LLM interaction data containing situations where LLMs fail to initiate grounding. We note that current frontier models perform poorly on Rifts, highlighting the need to reconsider how we train and prompt LLMs for human interaction. To this end, we develop a preliminary intervention aimed at mitigating grounding failures.

人机对话对话理解大模型评测接地机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。