为吉兹语系语言设计了兼顾形态边界的分词方法，提升语言保真度。
MoVoC: Morphology-Aware Subword Construction for Geez Script Languages
- 融合形态分析与BPE，构建形态感知的子词词汇表。
- 在4种吉兹语系语言上构建手动标注的形态数据集。
- 提升形态得分和边界精度，适合低资源复杂语言研究。
基于子词的分词方法常无法保留形态边界，这一问题在低资源、形态复杂的吉兹语系语言中尤为突出。为此，我们提出MoVoC（形态感知子词词汇构建）方法，并训练了集成监督形态分析的MoVoC-Tok分词器。该混合分段策略结合基于形态素和字节对编码（BPE）的标记，以在保持词汇语义的同时维护形态完整性。为应对资源匮乏问题，我们手工标注并发布了四种吉兹语系语言的形态素数据集，以及其中两种语言的形态感知词汇表。尽管所提分词方法未显著提升自动翻译质量，但在内在指标上表现出一致改进：形态得分（MorphoScore）和边界精确度（Boundary Precision）均提高，凸显形态感知分词在增强语言保真度和分词效率方面的价值。我们公开发布形态标注数据集和分词器，以支持低资源、形态丰富的语言研究。代码与数据已开源至GitHub：https://github.com/hailaykidu/MoVoC
原文摘要 · Abstract (English)
Subword-based tokenization methods often fail to preserve morphological boundaries, a limitation especially pronounced in low-resource, morphologically complex languages such as those written in the Geez script. To address this, we present MoVoC (Morpheme-aware Subword Vocabulary Construction) and train MoVoC-Tok, a tokenizer that integrates supervised morphological analysis into the subword vocabulary. This hybrid segmentation approach combines morpheme-based and Byte Pair Encoding (BPE) tokens to preserve morphological integrity while maintaining lexical meaning. To tackle resource scarcity, we curate and release manually annotated morpheme data for four Geez script languages and a morpheme-aware vocabulary for two of them. While the proposed tokenization method does not lead to significant gains in automatic translation quality, we observe consistent improvements in intrinsic metrics, MorphoScore, and Boundary Precision, highlighting the value of morphology-aware segmentation in enhancing linguistic fidelity and token efficiency. Our morpheme-annotated datasets and tokenizer will be publicly available to support further research in low-resource, morphologically rich languages. Our code and data are available on GitHub: https://github.com/hailaykidu/MoVoC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。