用字节级模型恢复克什米尔语缺失的变音符号，提升文本清晰度。
Koshur Diacritizer: A Byte-Level Sequence-to-Sequence Model for Kashmiri Diacritic Restoration
- 基于ByT5-small的字节级序列模型，直接处理原始字符
- 在23.7k对数据上达到0.2012的字符错误率
- 专为低资源语言设计，适合克什米尔语NLP研究者
克什米尔语是使用改良波斯-阿拉伯字母书写的印度-雅利安语，数字化文本中常省略变音符号，导致歧义并阻碍下游自然语言处理应用。我们提出Koshur Diacritizer，一个基于ByT5-small的字节级序列到序列模型，用于恢复克什米尔语文本中的变音符号。为此，我们公开发布了一个包含23.7k对齐的无变音/有变音克什米尔语句子对的数据集。所提框架结合了书写系统感知的归一化、对齐验证和骨架保持推理，确保可靠还原的同时保留原始基础字母序列。在预留测试集上的实验结果达到0.2012的字符错误率（DERm）和0.2159的词错误率（WER）。此外，母语克什米尔语语言学专家评估的平均准确率为77.5%。数据集、模型与源代码已公开，为克什米尔语变音符号恢复及未来低资源语言研究提供可复现基准。
原文摘要 · Abstract (English)
Kashmiri, an Indo-Aryan language written in a modified Perso-Arabic script, frequently omits diacritic marks in digital text, creating ambiguity and challenging downstream NLP applications. We present Koshur Diacritizer, a ByT5-small byte-level sequence-to-sequence model for restoring diacritics in Kashmiri text. To support this task, we release a publicly available dataset of 23.7k aligned undiacritized diacritized Kashmiri sentence pairs. The proposed framework combines script-aware normalization, alignment validation, and skeleton-preserving inference to ensure reliable restoration while maintaining the original base-letter sequence. Experimental results on a held-out test set achieve a DERm of 0.2012 and a WER of 0.2159. Additionally, evaluation by a native Kashmiri linguistic expert yields a mean accuracy of 77.5%. The dataset, model, and source code are publicly released to provide a reproducible baseline for Kashmiri diacritic restoration and future low-resource language research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。