用意大利语和英语的从句歧义句，测试大模型是否像人一样选句法偏好。
Investigating Syntactic Biases in Multilingual Transformers with RC Attachment Ambiguities in Italian and English
- 用从句附着歧义句测试模型句法偏好
- 模型普遍无法正确模仿人类偏好
- 适合研究跨语言模型语言知识与偏见
本文借鉴以往句子处理研究，探究单语和多语大语言模型在面对意大利语和英语相对从句附着歧义时，是否表现出类似人类的偏好。同时，我们检验了词汇因素（主句中的动词/名词类型）能否调节这些偏好，而这些因素已被证明与句法和语义关系的细微约束相关。总体结果表明，不同模型的行为差异显著，但普遍未能准确捕捉人类偏好。基于此，我们认为从句附着是跨语言评估大模型语言知识与偏见的理想基准。
原文摘要 · Abstract (English)
This paper leverages past sentence processing studies to investigate whether monolingual and multilingual LLMs show human-like preferences when presented with examples of relative clause attachment ambiguities in Italian and English. Furthermore, we test whether these preferences can be modulated by lexical factors (the type of verb/noun in the matrix clause) which have been shown to be tied to subtle constraints on syntactic and semantic relations. Our results overall showcase how LLM behavior varies interestingly across models, but also general failings of these models in correctly capturing human-like preferences. In light of these results, we argue that RC attachment is the ideal benchmark for cross-linguistic investigations of LLMs' linguistic knowledge and biases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。