解决不同分词器模型间概率评分难题，助力小模型高效学习大模型知识。
Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation
- 利用BPE算法的递归结构，构建跨分词器的概率计算框架。
- 在子集场景下实现精确概率计算，仅需每次采样一次模型推理。
- 显著降低模型内存占用（最多减少12%），提升推理准确率（最高+4%）。
在知识蒸馏等训练范式中，计算两个语言模型之间的下一个词概率比是标准任务。然而，当教师和学生模型使用不同分词器时，由于概率空间不一致，该任务变得困难，例如为降低边缘设备内存开销而采用更小词汇表的情况。本文揭示了常用字节对编码（BPE）算法中的隐式递归结构，并据此构建了一个跨分词器概率评分的统计框架。该方法可对与教师模型原生分词器不同的词汇表进行序列似然评估，涵盖两种情形：一是学生词汇表为教师词汇表的子集，二是任意词汇表情况。在子集情形下，框架能精确计算似然值，且每词仅需${O}(1)$次模型调用即可获得下一步概率用于序列采样。应用于蒸馏时，使Qwen2.5-1.5B模型内存占用最多减少12%，同时在评测任务上性能提升达4%。对于一般情形，提出一种严格无损的计算过程，结合快速近似方法，使大词汇量场景仍具实用性。在GSM8K数学推理蒸馏任务中，准确率超过现有最佳水平2%以上。代码已开源：github.com/truongbuu/cross-tokenizer-scoring。
原文摘要 · Abstract (English)
Computing next-token likelihood ratios between two language models (LMs) is a standard task in training paradigms such as knowledge distillation. Since this requires both models to share the same probability space, it becomes challenging when the teacher and student LMs use different tokenizers, for instance, when edge-device deployment necessitates a smaller vocabulary size to lower memory overhead. This work addresses this vocabulary misalignment problem by uncovering an implicit recursive structure in the commonly deployed Byte-Pair Encoding (BPE) algorithm and utilizing it to create a probabilistic framework for cross-tokenizer likelihood scoring. Our method enables sequence likelihood evaluation for vocabularies different from the teacher model native tokenizer, addressing two specific scenarios: when the student vocabulary is a subset of the teacher vocabulary, and the general case where it is arbitrary. In the subset regime, our framework computes exact likelihoods and provides next-token probabilities for sequential sampling with only ${O}(1)$ model evaluations per token. When used for distillation, this yields up to a $12\%$ reduction in memory footprint for the Qwen2.5-1.5B model while also improving baseline performance up to $4\%$ on the evaluated tasks. For the general case, we introduce a rigorous lossless procedure that leverages BPE recursive structure, complemented by a fast approximation that keeps large-vocabulary settings practical. Applied to GSM8K mathematical reasoning distillation, our method improves accuracy by over $2\%$ the current state of the art. Code: github.com/truongbuu/cross-tokenizer-scoring
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。