用回归模型预测最优语料混合比例，提升多语言分词器效率
TREX: Tokenizer Regression for Optimal Data Mixture
- 基于小规模代理分词器的压缩统计，建立数据混合与压缩性能的回归预测模型
- 在分布内和分布外测试中，压缩效率比LLaMA3和均匀分布混合提升最多12%
- 适合需要高效设计多语言分词器的研究者和工程团队
构建高效的多语言大模型分词器需精细控制语言特定语料比例。分词器的压缩性能直接影响大模型训练与推理效率，但现有方法依赖启发式或高成本的大规模搜索来确定最优语言比例。本文提出分词器回归框架TREX，通过在随机混合语料上训练小型代理分词器，收集其压缩统计信息，学习从语料混合比例预测压缩性能。该模型可实现大规模混合比例搜索前的高效预测，缓解多语言分词器设计中的精度-成本权衡。使用TREX预测混合比例训练的分词器，在分布内与分布外压缩效率上均比基于LLaMA3和均匀分布的混合提升最高达12%，展现出强可扩展性、鲁棒性和实际有效性。
原文摘要 · Abstract (English)
Building effective tokenizers for multilingual Large Language Models (LLMs) requires careful control over language-specific data mixtures. While a tokenizer's compression performance critically affects the efficiency of LLM training and inference, existing approaches rely on heuristics or costly large-scale searches to determine optimal language ratios. We introduce Tokenizer Regression for Optimal Data MiXture (TREX), a regression-based framework that efficiently predicts the optimal data mixture for tokenizer training. TREX trains small-scale proxy tokenizers on random mixtures, gathers their compression statistics, and learns to predict compression performance from data mixtures. This learned model enables scalable mixture search before large-scale tokenizer training, mitigating the accuracy-cost trade-off in multilingual tokenizer design. Tokenizers trained with TReX's predicted mixtures outperform mixtures based on LLaMA3 and uniform distributions by up to 12% in both inand out-of-distribution compression efficiency, demonstrating strong scalability, robustness, and practical effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。