测试发现多数大模型零样本句法分析能力不如基础基准，效果有限。
Better Benchmarking LLMs for Zero-Shot Dependency Parsing
- 用无输入信息的基线（如随机投影树）作为对比，评估大模型零样本句法解析能力
- 多数大模型表现不及最优基线，仅最新最大的LLaMA在多数语言上略胜一筹
- 揭示当前开源大模型在零样本句法解析上仍有明显不足，适合研究语言理解瓶颈者关注
尽管大语言模型在零样本任务中表现出色，但在句法解析等语言学挑战中的表现仍缺乏充分评估。本文通过与不依赖输入句子的基线（包括未在此类场景使用过的随机投影树和最优线性排列）比较，考察了最先进的开源大模型在该任务上的表现。结果表明，大多数测试的大模型无法超越最佳无信息基线，仅有最新且最大的LLaMA版本在多数语言上取得优势，但仍达到较低水平。因此，当前开源大模型尚无法实现准确的零样本句法解析。
原文摘要 · Abstract (English)
While LLMs excel in zero-shot tasks, their performance in linguistic challenges like syntactic parsing has been less scrutinized. This paper studies state-of-the-art open-weight LLMs on the task by comparing them to baselines that do not have access to the input sentence, including baselines that have not been used in this context such as random projective trees or optimal linear arrangements. The results show that most of the tested LLMs cannot outperform the best uninformed baselines, with only the newest and largest versions of LLaMA doing so for most languages, and still achieving rather low performance. Thus, accurate zero-shot syntactic parsing is not forthcoming with open LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。