现有分词方式削弱语言模型对语音特征的表达，新方法可有效改善。
How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them
- 用音素符号微调模型，让其更关注语音结构
- 分词与音节边界越不匹配，语音表征越差
- 提升语音相关任务表现，同时保持推理能力
分词是语言模型的第一步，但从未考虑词语发音。我们通过探测实验发现，基于子词的分词方式系统性地弱化了局部（如押韵）和全局（如音节划分）语音特征的编码。为此，我们提出一种衡量分词与自然音节边界错位程度的指标——音节-分词对齐距离（STAD），并发现错位越高，语音表征越差，可作为语音感知分词的诊断工具。为解决此问题，我们提出一种轻量级的基于国际音标（IPA）的微调方法，使语言模型具备语音意识，在三项语音相关任务中均取得一致提升，同时在GSM8K和MMLU上仅分别下降1.1%和0.9%，基本保持数学与通用推理能力。
原文摘要 · Abstract (English)
Tokenization is the first step in every language model (LM), yet it never takes the sounds of words into account. We investigate how tokenization influences text-only LMs' ability to represent phonological knowledge. Through a series of probing experiments, we show that subword-based tokenization systematically weakens the encoding of both local (e.g., rhyme) and global (e.g., syllabification) phonological features. To quantify this effect, we introduce the syllabification-tokenization alignment distance (STAD), a metric that measures the misalignment between a model's tokenization and the natural syllable boundaries of words, and find that higher misalignment correlates with poorer phonological representations, providing a simple diagnostic for phonology-aware tokenization. To address these limitations, we propose a lightweight IPA-based fine-tuning method that infuses phonological awareness into LMs, leading to consistent improvements across three phonology-related tasks while largely preserving math and general reasoning ability, with 1.1\% and 0.9\% drops on GSM8K and MMLU, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。