中文语义模糊下大模型信任度脆弱，易误判多义文本
Uncovering the Fragility of Trustworthy LLMs through Chinese Textual Ambiguity
- 构建中文语义模糊数据集，涵盖3类9子类歧义句对
- 模型无法区分模糊与非模糊文本，误判率超人类水平
- 适合关注语言理解可靠性、AI安全的研究者
本文研究大语言模型在处理语义模糊文本时的可信度问题，聚焦中文语境。我们通过收集和生成带上下文的模糊句子及其对应的消歧句对，构建了一个基准数据集，系统划分为3大类9小类。实验发现，当前大模型在处理模糊文本时存在显著脆弱性：无法可靠区分模糊与非模糊文本；过度自信地将多义文本视为单义；在尝试理解多种可能含义时表现出过度思考。这些结果揭示了现有模型在语言理解中对不确定性的处理缺陷，对实际应用具有重要影响。代码与数据集已公开于GitHub。
原文摘要 · Abstract (English)
In this work, we study a critical research problem regarding the trustworthiness of large language models (LLMs): how LLMs behave when encountering ambiguous narrative text, with a particular focus on Chinese textual ambiguity. We created a benchmark dataset by collecting and generating ambiguous sentences with context and their corresponding disambiguated pairs, representing multiple possible interpretations. These annotated examples are systematically categorized into 3 main categories and 9 subcategories. Through experiments, we discovered significant fragility in LLMs when handling ambiguity, revealing behavior that differs substantially from humans. Specifically, LLMs cannot reliably distinguish ambiguous text from unambiguous text, show overconfidence in interpreting ambiguous text as having a single meaning rather than multiple meanings, and exhibit overthinking when attempting to understand the various possible meanings. Our findings highlight a fundamental limitation in current LLMs that has significant implications for their deployment in real-world applications where linguistic ambiguity is common, calling for improved approaches to handle uncertainty in language understanding. The dataset and code are publicly available at this GitHub repository: https://github.com/ictup/LLM-Chinese-Textual-Disambiguation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。