测试大模型对中文反身代词ziji的远距离绑定理解能力
Language Models at the Syntax-Semantics Interface: A Case Study of the Long-Distance Binding of Chinese Reflexive ziji
- 构建240个合成句+320个真实句数据集，评估模型对语法语义约束的把握
- 所有模型均未达到母语者水平，普遍依赖序列线索而忽略深层约束
- 模型更关注名词语义而非动词语义，暴露对复杂语言结构的理解短板
本文探究大语言模型是否能有效处理汉语反身代词ziji的复杂绑定模式，该模式受语法和语义双重制约。研究构建了包含240个合成句子（基于句法文献模板）和320个来自BCC语料库的真实句子的数据集。评估21个语言模型在该数据集上的表现，并与母语者判断进行对比。结果显示，现有模型均未能一致复现人类判断。结果表明，现有模型过度依赖序列线索，但并不总偏好最近字符串，且常忽视细微的语义与句法约束。它们对名词相关语义更为敏感，而对动词相关语义反应较弱。
原文摘要 · Abstract (English)
This paper explores whether language models can effectively resolve the complex binding patterns of the Mandarin Chinese reflexive ziji, which are constrained by both syntactic and semantic factors. We construct a dataset of 240 synthetic sentences using templates and examples from syntactic literature, along with 320 natural sentences from the BCC corpus. Evaluating 21 language models against this dataset and comparing their performance to judgments from native Mandarin speakers, we find that none of the models consistently replicates human-like judgments. The results indicate that existing language models tend to rely heavily on sequential cues, though not always favoring the closest strings, and often overlooking subtle semantic and syntactic constraints. They tend to be more sensitive to noun-related than verb-related semantics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。