为新领域定制分词器，让大模型更高效地处理蛋白质序列。
Adapting Language Models via Token Translation
- 用稀疏Sinkhorn算法训练目标领域的专属分词器并学习跨域词元转换。
- 在蛋白质序列上降低困惑度，压缩率优于直接微调源或目标分词器。
- 小模型学的翻译可迁移至大模型，低成本获得更好性能。
现代大语言模型使用固定分词器对源领域文本进行高效压缩，但在新目标领域应用时会导致压缩效果下降、推理成本增加及语义对齐减弱。为此，我们提出稀疏Sinkhorn词元翻译（S2T2），为目标领域训练专用分词器，并学习目标与源词元间的映射，从而更有效复用预训练的下一个词元预测器。在微调后的英文语言模型上，S2T2在域外蛋白质序列上同时提升了困惑度和压缩效率，优于使用源或目标分词器的直接微调。此外，我们发现为小型、低成本模型学习的词元翻译可直接迁移到大型、高性能模型，以更低代价获得S2T2收益。
原文摘要 · Abstract (English)
Modern large language models use a fixed tokenizer to effectively compress text drawn from a source domain. However, applying the same tokenizer to a new target domain often leads to inferior compression, more costly inference, and reduced semantic alignment. To address this deficiency, we introduce Sparse Sinkhorn Token Translation (S2T2). S2T2 trains a tailored tokenizer for the target domain and learns to translate between target and source tokens, enabling more effective reuse of the pre-trained next-source-token predictor. In our experiments with finetuned English language models, S2T2 improves both the perplexity and the compression of out-of-domain protein sequences, outperforming direct finetuning with either the source or target tokenizer. In addition, we find that token translations learned for smaller, less expensive models can be directly transferred to larger, more powerful models to reap the benefits of S2T2 at lower cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。