语言模型不遵守语法规则，可能反而是其优势。
Linguistic Generalizations are not Rules: Impacts on Evaluation of LMs
- 提出自然语言并非由严格规则构成，而是依赖灵活语境的组合
- 发现语言模型对非规则语言现象的处理能力优于传统规则评估
- 适合研究语言本质与模型泛化能力的学者参考
当前语言模型的语言能力评估常基于自然语言由符号规则生成的假设：语法正确性取决于是否符合规则，语义解析通过句法规则组合词义实现。然而，本文指出自然语言并非由清晰分离、组合式的规则构成，新表达的产生与理解依赖于灵活、关联且情境依赖的结构组合。语言模型未能遵循严格规则，未必是缺陷，反而可能反映其对真实语言中梯度因素（如频率、上下文、功能）的捕捉能力。这提示我们需要重新设计评估基准与分析方法，以更准确探测模型对自然语言丰富灵活性的掌握程度。
原文摘要 · Abstract (English)
Linguistic evaluations of how well LMs generalize to produce or understand language often implicitly take for granted that natural languages are generated by symbolic rules. According to this perspective, grammaticality is determined by whether sentences obey such rules. Interpretation is compositionally generated by syntactic rules operating on meaningful words. Semantic parsing maps sentences into formal logic. Failures of LMs to obey strict rules are presumed to reveal that LMs do not produce or understand language like humans. Here we suggest that LMs' failures to obey symbolic rules may be a feature rather than a bug, because natural languages are not based on neatly separable, compositional rules. Rather, new utterances are produced and understood by a combination of flexible, interrelated, and context-dependent constructions. Considering gradient factors such as frequencies, context, and function will help us reimagine new benchmarks and analyses to probe whether and how LMs capture the rich, flexible generalizations that comprise natural languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。