评测大模型对韩语句尾的掌握，发现提示缺失句尾能提升表现
Making Sense of Korean Sentences: A Comprehensive Evaluation of LLMs through KoSEnd Dataset
- 构建3000句韩语句尾数据集，标注15种结尾形式自然度
- 11个大模型在句尾判断上表现差异大，参数量与一致性相关
- 提示可能缺少句尾可显著提升模型准确率，适合低资源语言研究者
尽管大模型在多种语言中取得进展，但其在低资源黏着语（如韩语）上的表现仍存疑。本研究聚焦韩语复杂句尾结构，构建了韩国句尾（KoSEnd）数据集，包含3000条句子，每句标注15种句尾形式的自然度，来源多样以覆盖不同语境。评估了11个大模型对韩语句尾的理解能力，按参数量和预测一致性分析。结果发现，提示模型可能存在缺失句尾的情况，能显著提升性能，凸显明确考虑特定语言特征的重要性。
原文摘要 · Abstract (English)
Although LLMs have made significant progress in various languages, there are still concerns about their effectiveness with low-resource agglutinative languages compared to languages such as English. In this study, we focused on Korean, a language known for its complex sentence endings, and evaluated LLMs on this challenging aspect. We introduce the Korean Sentence Endings (KoSEnd) dataset, which includes 3,000 sentences, each annotated for the naturalness of 15 sentence ending forms. These were collected from diverse sources to cover a range of contexts. We evaluated 11 LLMs to assess their understanding of Korean sentence endings, analyzing them based on parameter count and prediction consistency. Notably, we found that informing models about the possibility of missing sentence endings improved performance, highlighting the impact of explicitly considering certain linguistic features.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。