通过同形词增强训练，延缓语言模型过拟合。
Training Language Models with homotokens Leads to Delayed Overfitting
- 用同形词替代分词作为数据增强，保持语义不变
- 在数据受限时显著延迟过拟合，提升泛化能力
- 适合追求鲁棒性与跨数据集性能的模型优化
子词分词在语言模型中引入了计算层，多个不同的标记序列可解码为相同表面形式且语义一致，但引发不同内部计算。尽管存在非唯一性，语言模型通常使用单一标准最长前缀分词进行训练。本文将同形词——同一词汇项的替代有效子词分割——形式化为严格语义保留的数据增强方式。提出一种轻量级训练架构，通过辅助因果编码器和块因果交叉注意力，使标准下一个词预测依赖于采样的同形词变体，不改变训练目标或标记接口。在数据受限的预训练中，同形词增强在重复数据暴露下持续延缓过拟合，并在多种评估数据集上提升泛化性能。多语言微调中发现，同形词效果取决于分词器质量：当标准分词高度压缩时增益最强，分词已过度碎片化时则减弱。总体而言，同形词为语言模型提供了一种简单、模块化的分词不变性机制。
原文摘要 · Abstract (English)
Subword tokenization introduces a computational layer in language models where many distinct token sequences decode to the same surface form and preserve meaning, yet induce different internal computations. Despite this non-uniqueness, language models are typically trained using a single canonical longest-prefix tokenization. We formalize homotokens-alternative valid subword segmentations of the same lexical item-as a strictly meaning-preserving form of data augmentation. We introduce a lightweight training architecture that conditions canonical next-token prediction on sampled homotoken variants via an auxiliary causal encoder and block-causal cross-attention, without modifying the training objective or token interface. In data-constrained pretraining, homotoken augmentation consistently delays overfitting under repeated data exposure and improves generalization across diverse evaluation datasets. In multilingual fine-tuning, we find that the effectiveness of homotokens depends on tokenizer quality: gains are strongest when canonical tokens are highly compressed and diminish when the tokenizer already over-fragments the input. Overall, homotokens provide a simple and modular mechanism for inducing tokenization invariance in language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。