为土耳其语设计的神经形态分词器，可无损还原原始文本并生成结构化词向量。
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
- 基于字符边界概率的可微动态规划，实现软分词与精确切分。
- 比特率低至1.425，形态对齐准确率提升至0.61，显存占用减少19%。
- 适合需要精确分词与词根表征的土耳其语自然语言任务。
土耳其语是黏着语，意义由词素承载，但现代语言模型常用的子词分词器依据语料统计切分，破坏了语义丰富的后缀；而WordPiece和规则分析器无法还原原始文本。本文提出Morpheus，一种神经形态边界模型，既是无损、形态感知的分词器，也是词向量生成器。通过可微泊松-二项式动态规划，训练时生成软词素归属，推理时精确分割，无需字符串归一化，保证解码等于原词（decode(encode(w)) = w）。模型前向传播同时输出结构化词向量。在可逆分词器中，Morpheus比特率最低（1.425），形态对齐得分（MorphScore macro-F1）达0.61，显著优于子词族的约0.32，且比64K词表子词分词器节省约19%显存。作为嵌入器，冻结的Morpheus向量在词根检索（根族MAP 0.85）和同根验证（ROC-AUC 1.00）上超越BGE-M3与BERTurk；但在依赖上下文和变位的任务（如命名实体识别、格/数探测）上，仍落后于更复杂的上下文编码器——这归因于其根心几何结构。
原文摘要 · Abstract (English)
Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text. This paper presents \textbf{Morpheus}, a neural morpheme-boundary model for Turkish that is at once a lossless, morphology-aware tokenizer and a word-embedding producer. A differentiable Poisson-binomial dynamic program turns per-character boundary probabilities into soft morpheme memberships during training and exact segments at inference, with no string normalization, so $\mathrm{decode}(\mathrm{encode}(w)) = w$ holds by construction. Because the model is neural, the same forward pass that tokenizes also emits a structured word embedding. Among reversible tokenizers -- the only ones valid for generation -- Morpheus attains the lowest bits-per-character ($1.425$), roughly doubles the gold morphological alignment of the subword family (MorphScore macro-F1 $0.61$ vs.\ ${\sim}0.32$), and uses ${\sim}19\%$ less GPU memory than 64K-vocabulary subword tokenizers. As an embedder, frozen Morpheus vectors lead on lexical retrieval (root-family MAP $0.85$) and same-root verification (ROC-AUC $1.00$), surpassing the multilingual retriever BGE-M3 and BERTurk; on context- and inflection-dependent tasks (NER, case/number probing) the heavier contextual encoders remain ahead -- a trade-off we attribute to Morpheus's root-centric geometry. Code: https://github.com/lonewolf-rd/TurkishMorpheus; model: https://huggingface.co/lonewolflab/Morpheus-TR-50K; interactive demo: https://huggingface.co/spaces/lonewolflab/morpheus-tr-demo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。