让大模型适应新语言时,用注意力机制优化分词器初始化。
Model-Aware Tokenizer Transfer
- 通过注意力行为建模,指导新分词器的嵌入初始化。
- 几小时训练后恢复原模型大部分性能,超越传统方法。
- 适合多语言模型适配低资源语言或新文字系统。
大型语言模型(LLMs)支持的语言越来越多,但其预设分词器仍是适配低资源或不同书写系统语言的瓶颈。现有分词器迁移方法依赖语义启发式初始化嵌入,忽略模型高层动态,限制迁移质量。我们提出模型感知分词器迁移(MATT),将模型内部信息融入迁移过程。MATT引入注意力影响建模(AIM)目标,从源模型中提炼跨标记的交互模式,并注入目标模型的新分词器中,实现高效预热。与仅关注嵌入相似性的方法不同,MATT利用注意力行为引导嵌入初始化与调整。在多种语言环境下实验表明,MATT在数小时GPU时间内恢复了原始模型的大量性能,优于启发式基线。结果表明,引入模型级信号是实现多语言LLM鲁棒分词器迁移的有效路径。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are trained to support an increasing number of languages, yet their predefined tokenizers remain a bottleneck for adapting models to lower-resource or distinct-script languages. Existing tokenizer transfer methods typically rely on semantic heuristics to initialize new embeddings, ignoring higher-layer model dynamics and limiting transfer quality. We propose Model-Aware Tokenizer Transfer (MATT), a method that incorporates model internals into the tokenizer transfer process. MATT introduces an Attention Influence Modeling (AIM) objective that distills inter-token communication patterns from a source model into a target model with a new tokenizer, providing an efficient warm-up before standard language modeling. Unlike approaches that focus solely on embedding similarity, MATT leverages attention behavior to guide embedding initialization and adaptation. Experiments across diverse linguistic settings show that MATT recovers a large fraction of the original model's performance within a few GPU hours, outperforming heuristic baselines. These results demonstrate that incorporating model-level signals offers a practical and effective path toward robust tokenizer transfer in multilingual LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。