用音节切分法解决土耳其语检索的词汇外问题，效果优于传统方法。
HeceTokenizer: A Syllable-Based Tokenization Approach for Turkish Retrieval
- 基于土耳其语音节的六种确定性模式构建8000个音节词表，实现零词汇外。
- 在TQuAD上达到50.3%的Recall@5，超越200倍大的基线模型。
- 适合需要低资源、高鲁棒性检索的土耳其语应用。
HeceTokenizer 是一种面向土耳其语的音节切分分词器，利用该语言确定性的六种音节结构，构建了一个约8000个唯一音节类型的封闭词表，实现无词汇外（OOV）问题。采用150万参数的BERT-tiny编码器在土耳其语维基百科子集上从头训练，以掩码语言建模为目标，在TQuAD检索基准上评估，结合细粒度块检索策略后，取得50.3%的Recall@5。这一结果超过使用200倍更大模型的形态驱动基线模型的46.92%。结果表明，土耳其语音节的音韵规律为检索任务提供了强大且轻量的归纳偏置。
原文摘要 · Abstract (English)
HeceTokenizer is a syllable-based tokenizer for Turkish that exploits the deterministic six-pattern phonological structure of the language to construct a closed, out-of-vocabulary (OOV)-free vocabulary of approximately 8,000 unique syllable types. A BERT-tiny encoder (1.5M parameters) is trained from scratch on a subset of Turkish Wikipedia using a masked language modeling objective and evaluated on the TQuAD retrieval benchmark using Recall@5. Combined with a fine-grained chunk-based retrieval strategy, HeceTokenizer achieves 50.3% Recall@5, surpassing the 46.92% reported by a morphology-driven baseline that uses a 200 times larger model. These results suggest that the phonological regularity of Turkish syllables provides a strong and resource-light inductive bias for retrieval tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。