测试大模型对爪哇语敬语系统的理解能力，发现表现普遍不佳且有偏向。
Do Language Models Understand Honorific Systems in Javanese?
- 构建爪哇语敬语专用数据集Unggah-Ungguh，涵盖社会层级与语境的用词差异。
- 大模型在敬语分类和翻译任务中表现差，多数层级识别准确率低于60%。
- 适合研究低资源语言、社会语用学与文化敏感性自然语言处理的学者。
爪哇语具有复杂的敬语系统，其词汇选择随说话者、听者及提及对象的社会地位而变化。尽管该系统具有重要的文化和语言意义，但针对自然语言处理任务的全面语料库仍十分有限。本文提出Unggah-Ungguh，一个精心构建的数据集，用于捕捉爪哇语社交礼仪体系Unggah-Ungguh Basa中的细微差别，该体系依据社会等级和语境决定用词。利用该数据集，我们通过分类和机器翻译任务评估语言模型（LMs）对不同敬语层级的处理能力。为评估跨语言模型表现，还进行了爪哇语（特定敬语层级）与印尼语之间的机器翻译实验。此外，我们探索了模型在对话任务中生成符合语境的爪哇语敬语的能力，要求其根据社会角色和上下文线索合理选择敬语形式。结果表明，当前语言模型在大多数敬语层级上表现不佳，且存在对某些敬语层级的系统性偏好。
原文摘要 · Abstract (English)
The Javanese language features a complex system of honorifics that vary according to the social status of the speaker, listener, and referent. Despite its cultural and linguistic significance, there has been limited progress in developing a comprehensive corpus to capture these variations for natural language processing (NLP) tasks. In this paper, we present Unggah-Ungguh, a carefully curated dataset designed to encapsulate the nuances of Unggah-Ungguh Basa, the Javanese speech etiquette framework that dictates the choice of words and phrases based on social hierarchy and context. Using Unggah-Ungguh, we assess the ability of language models (LMs) to process various levels of Javanese honorifics through classification and machine translation tasks. To further evaluate cross-lingual LMs, we conduct machine translation experiments between Javanese (at specific honorific levels) and Indonesian. Additionally, we explore whether LMs can generate contextually appropriate Javanese honorifics in conversation tasks, where the honorific usage should align with the social role and contextual cues. Our findings indicate that current LMs struggle with most honorific levels, exhibitinga bias toward certain honorific tiers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。