测试11个Transformer模型对人类句子处理的预测能力，发现其在复杂句式中表现不佳。
Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects
- 用困惑度机制评估模型对英语语法吸引效应的预测
- 在宾语提取从句中模型预测与人类阅读时间差异显著
- 不同模型间预测分歧大，无法复现人类不对称干扰模式
Transformer模型广泛应用于自然语言处理，但其作为人类句子处理认知模型的有效性仍存争议。本文采用基于困惑度的关联机制，系统评估了11个不同规模和结构的自回归Transformer模型，在比以往研究更全面的英语语法吸引配置下进行测试。实验结果混合：尽管模型在介词短语结构上预测与人类阅读时间数据基本一致，但在宾语提取的定语从句结构中性能显著下降。在此类结构中，模型间的预测差异明显，且无一模型能成功复现人类观察到的不对称干扰模式。结论指出，当前Transformer模型无法解释人类形态句法处理机制，评估其作为认知模型时必须采用严谨、全面的实验设计，避免因孤立句型或单一模型得出误导性结论。
原文摘要 · Abstract (English)
Transformers underlie almost all state-of-the-art language models in computational linguistics, yet their cognitive adequacy as models of human sentence processing remains disputed. In this work, we use a surprisal-based linking mechanism to systematically evaluate eleven autoregressive transformers of varying sizes and architectures on a more comprehensive set of English agreement attraction configurations than prior work. Our experiments yield mixed results: While transformer predictions generally align with human reading time data for prepositional phrase configurations, performance degrades significantly on object-extracted relative clause configurations. In the latter case, predictions also diverge markedly across models, and no model successfully replicates the asymmetric interference patterns observed in humans. We conclude that current transformer models do not explain human morphosyntactic processing, and that evaluations of transformers as cognitive models must adopt rigorous, comprehensive experimental designs to avoid spurious generalizations from isolated syntactic configurations or individual models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。