仅凭公开分词器就能精准估算大模型训练数据组成。
Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?
- 用分位数趋势拟合词表中词元的分布规律。
- 在控制和真实场景下，词元级误差低至3.00%。
- 适合想逆向分析模型训练数据的研究者使用。
预训练语料库构成决定了大模型的能力，但通常在模型权重公开后仍不透明。以往研究通过公开分词器推断语料混合比例或追踪特定词元组；本文提出对任意目标词元估计其来源语料比例。我们发现，不同语料上训练的BPE分词器具有稳定的词元ID-比例分布，据此可将已知语料的分布迁移至未知语料训练的目标分词器。为此提出分位数引导密度估计（QGDE），通过多分位数趋势逼近分布，并利用局部密度加权实现词元级估计。在控制环境和使用公开SmolLM分词器的真实场景下，词元级估计平均相对误差低至3.00%，聚合为类别级混合比例后误差为3.08%。结果表明，公开分词器可提供细粒度语料估计的有效信号，超越粗粒度推断。
原文摘要 · Abstract (English)
Pretraining corpus composition shapes LLM capabilities, but it often remains hidden even when model weights are released. Prior work has inferred corpus mixtures or traced specific token groups from released tokenizer vocabularies; in contrast, we estimate corpus ratios for arbitrary target tokens. We first show that BPE tokenizers trained on different corpora share stable token ID--ratio distributions, motivating distribution transfer from known corpora to a target tokenizer trained on hidden corpora. We then propose Quantile-Guided Density Estimation (QGDE), which approximates this distribution with multiple quantile trends and uses local density weighting to produce token-level estimates. In controlled settings and a realistic setting using the released SmolLM tokenizer, QGDE achieves mean relative errors as low as 3.00% for token-level estimation and 3.08% after aggregation into category-level mixtures. These results suggest that released tokenizer vocabularies provide a useful signal for fine-grained corpus estimation beyond coarse composition inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。