测试大模型理解非语言对话意图的能力，发现表现远低于语言对话。
Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses
- 构建纯非语言对话数据集，评估大模型的语用推理能力。
- 非语言意图理解准确率比语言对话低60个百分点。
- 上下文学习可提升模型对非语言意图的捕捉能力。
尽管大语言模型（LLMs）在语用理解方面取得显著进展，但以往研究主要集中于其对言语行为的理解。然而，非言语行为仍是人类交流的核心组成部分，尤其在刻意孤立使用以传达间接含义时。本文首次系统评估了LLMs在仅包含非言语回应的对话中推断语用意义的能力。我们提出三个研究问题：（1）LLMs能否识别通过非言语行为传达的间接意图？（2）在何种情况下及如何失败？（3）如何提升其解释非言语意图的能力？评估结果显示，与言语情境相比，LLMs在非言语意图理解上的准确率下降高达60个百分点。进一步分析揭示了模型在解读非言语行为时的行为模式，并表明上下文学习有助于提升语用推理能力。
原文摘要 · Abstract (English)
Although large language models (LLMs) have shown considerable progress in pragmatic language understanding, prior research has focused mainly on their comprehension of verbal behavior. Nonetheless, non-verbal behavior remains a fundamental component of human communication, especially when deliberately utilized in isolation to convey indirect meanings. In this work, we present the first systematic evaluation of LLMs' ability to infer pragmatic meaning in dialogue consisting solely of non-verbal responses. We explore three research questions: (1) Can LLMs recognize indirect intent conveyed through non-verbal responses? (2) When and how do LLMs fail to capture non-verbal intent? (3) How can we improve LLMs' ability to interpret non-verbal intent?. Through the evaluation, we observe that LLMs struggle to infer underlying meaning from non-verbal responses, with accuracy dropping by up to 60% points compared to verbal ones. Further extensive analysis reveals a behavioral pattern in LLMs' interpretations of non-verbal behavior and demonstrates that in-context learning facilitates pragmatic inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。