测试多语言模型对词序与词形的依赖，发现不同语言差异显著。
A Typologically Grounded Evaluation Framework for Word Order and Morphology Sensitivity in Multilingual Masked LMs
- 通过扰动词序和词形，评估模型对语言结构的敏感性。
- 中文受词序影响小，德西俄语则严重依赖词形变化。
- 适合研究多语言模型语言特性或跨语言泛化能力者阅读。
我们提出一种基于语言类型学的诊断框架，用于评估多语言掩码语言模型对词序与屈折形态的依赖程度。利用通用依存库（Universal Dependencies），在推理时施加四种扰动：完整词元打乱、保留功能词的实词打乱、基于依存关系的头-修饰词互换，以及句级词干替换（+L），即同时对上下文和被掩码目标词进行词干化处理。在英语、汉语、德语、西班牙语和俄语上评估mBERT和XLM-R。完整打乱使所有语言的词级重建准确率趋近于零；部分扰动和头-修饰词互换也导致显著下降。+L在汉语中影响极小，但在德语、西班牙语和俄语中大幅降低准确率，且无法缓解打乱带来的影响。前5名预测准确率同样显示，在完全打乱下，正确词极少出现在前五名中。代码、采样脚本及平衡评估子集已公开，土耳其语在严格重建下的结果见附录。
原文摘要 · Abstract (English)
We introduce a typology-aware diagnostic for multilingual masked language models that tests reliance on word order versus inflectional form. Using Universal Dependencies, we apply inference-time perturbations: full token scrambling, content-word scrambling with function words fixed, dependency-based head--dependent swaps, and sentence-level lemma substitution (+L), which lemmatizes both the context and the masked target label. We evaluate mBERT and XLM-R on English, Chinese, German, Spanish, and Russian. Full scrambling drives word-level reconstruction accuracy near zero in all languages; partial and head--dependent perturbations cause smaller but still large drops. +L has little effect in Chinese but substantially lowers accuracy in German/Spanish/Russian, and it does not mitigate the impact of scrambling. Top-5 word accuracy shows the same pattern: under full scrambling, the gold word rarely appears among the five highest-ranked reconstructions. We release code, sampling scripts, and balanced evaluation subsets; Turkish results under strict reconstruction are reported in the appendix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。