让语言模型在对话中主动请求澄清，测试其自我认知能力
Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests
- 用参考游戏任务检验模型识别自身不确定性的能力
- 三类视觉语言模型在不确定时请求澄清的准确率普遍偏低
- 适合研究人机交互中的不确定性对齐问题
在人类对话中，双方共同维护理解。当听者不确定说话人意思时，会主动请求澄清。当前一个开放问题是：语言模型能否扮演类似听者角色，通过澄清请求表达自身不确定性。我们提出，参考游戏是理想的测试场景——它控制性强、自包含，并使澄清需求显式且可度量。我们评估了三种视觉-语言模型，在基准参考解析任务与要求模型在不确定时主动请求澄清的实验任务之间进行对比。结果显示，即使在简单任务中，模型也普遍难以识别内部不确定性，也无法有效转化为恰当的澄清行为。这表明参考游戏作为测试语言与视觉语言模型交互质量的工具具有重要价值。
原文摘要 · Abstract (English)
In human conversation, both interlocutors play an active role in maintaining mutual understanding. When listeners are uncertain about what speakers mean, for example, they can request clarification. It is an open question for language models whether they can assume a similar listener role, recognizing and expressing their own uncertainty through clarification. We argue that reference games are a suitable testbed to approach this question as they are controlled, self-contained, and make clarification needs explicit and measurable. To test this, we evaluate three vision-language models comparing a baseline reference resolution task to an experiment where the models are instructed to request clarification when uncertain. The results suggest that even in such simple tasks, models often struggle to recognize internal uncertainty and translate it into adequate clarification behavior. This demonstrates the value of reference games as testbeds for interaction qualities of (vision and) language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。