为泰米尔语设计可逆语素分词与分层词组合，提升小模型翻译性能
Morphology Aware Reversible Semantic Tokenization and Hierarchical Word Composition for Tamil Language Models
- 基于语素分析的可逆分词，保留字节精确重建能力
- 分层词组合使序列长度减少59.3%，推理耗能降低9-21%
- 在小模型预算下显著优于现有基线，适合资源受限场景
统计子词分词器虽能处理任意文本，但其切分单位未必符合词汇或语法结构。这对泰米尔语尤为重要，因其书面词常包含词干变化、格、数、时态、一致、语态、附着词和连动词等信息。本文提出一个扩展开源ThamizhiMorph分析器与生成器的泰米尔语素系统，配合字节精确的语义分词器和学习型分层词组合器。十二个有限状态转换器将词分解为词根与语法特征，同时保留字符和字节级回退机制以实现精确重构。在相同69,591对泰米尔-英语训练数据、18.97万参数编码器-解码器、40,000次更新、统一目标分词器、优化器、位置编码方法及生成设置下，平坦语素分词器在保护的3,539行IN22和FLORES+测试集上取得最佳综合表现：BLEU 10.63，chrF++ 35.26，COMETKiwi 0.6276。相较最强外部基线AI4Bharat，分别提升7.2%、3.2%、2.6%。词组合器得分10.30、34.88、0.6241，提升3.8%、2.1%、2.0%。组合器将平均全局源状态从71.48降至29.08（降幅59.3%），预计推理所需浮点运算量减少9-21%（取决于解码缓存）。剩余性能差距集中在更长的FLORES+句子。结果表明，在固定小模型预算下，显式语素建模能提升翻译质量，而分层组合则大幅压缩序列长度并降低推理开销。
原文摘要 · Abstract (English)
Statistical subword tokenizers can process arbitrary text, but their units need not align with lexical or grammatical structure. This is especially important for Tamil, where a written word may encode stem changes, case, number, tense, agreement, voice, clitics, and linked verbs. We present a Tamil morphology system extending the open-source ThamizhiMorph analyzer and generator, together with a byte-exact semantic tokenizer and a learned hierarchical word composer. Twelve finite-state transducers analyze words into lemmas and grammatical features, while character and byte fallbacks preserve exact reconstruction. We compare a flat morphology tokenizer, a signal-preserving word composer, and tokenizers based on Sarvam-1, AI4Bharat IndicBERTv2, and BrahmicTokenizer-131K. All systems use the same 69,591 Tamil-English training pairs, 18.97-million-parameter encoder-decoder, 40,000 updates, target tokenizer, optimizer, positional method, and generation settings. On a protected 3,539-row IN22 and FLORES+ evaluation, morphology-flat achieves the best pooled scores: 10.63 BLEU, 35.26 chrF++, and 0.6276 COMETKiwi. Relative to AI4Bharat, the strongest external-tokenizer baseline, these are improvements of 7.2%, 3.2%, and 2.6%. The word composer scores 10.30, 34.88, and 0.6241, improving on AI4Bharat by 3.8%, 2.1%, and 2.0%. The composer reduces mean global source states from 71.48 to 29.08, a 59.3% reduction, and is estimated to require 9-21% fewer inference FLOPs depending on decoder caching. Its remaining quality gap is concentrated in longer FLORES+ sentences. These results show that explicit Tamil morphology improves translation under a fixed small-model budget, while hierarchical composition substantially reduces sequence length and estimated inference cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。