改进扩散语言模型的子词编码与索引打乱，提升小模型零样本推理能力
MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language Models
- 采用二进制编码与索引打乱优化子词粒度设计
- 1.1B参数模型在8个常识推理任务上平均零样本准确率领先
- 提供可量化的子词设计准则，适合大模型高效训练
掩码扩散模型（MDM）在使用部分遮蔽策略（Prime）时展现出更优的泛化能力。该方法将词元转换为子词元，并在子词元层面建模扩散过程。我们发现当前MDM-Prime框架存在两大局限：其一，子词生成器的函数形式与常用字节对编码（BPE）分词器结合时，显著增加目标函数的交叉熵损失；其二，缺乏指导子词粒度超参数选择的工具。为此，我们分析了最小化MDM-Prime训练目标的最优子词生成器设计，提出MDM-Prime-v2，引入二进制编码与索引打乱机制。我们的分析揭示了词元粒度与子词熵如何影响训练目标和下游性能，提供了子词生成器设计的理论依据。当模型规模扩展至1.1B参数时，MDM-Prime-v2在八个常识推理基准测试中展现出更高的平均零样本准确率，优于同等规模的GPT-Neo、OPT、Pythia、Bloom、SMDM和TinyLLaMA等基线模型。
原文摘要 · Abstract (English)
Masked diffusion models (MDM) exhibit superior generalization when learned using a Partial masking scheme (Prime). This approach converts tokens into sub-tokens and models the diffusion process at the sub-token level. We identify two limitations of the MDM-Prime framework. First, we find that the functional form of the subtokenizer significantly increases the cross-entropy loss in the objective when paired with commonly used Byte-Pair-Encoding (BPE) tokenizers. Second, we lack tools to guide the hyperparameter choice of the token granularity in the subtokenizer. To address these limitations, we analyze the optimal design of the subtokenizer that minimizes MDM-Prime training objective and develop MDM-Prime-v2, a masked diffusion language model which incorporates Binary Encoding and Index Shuffling. Our analysis characterizes how token granularity and sub-token entropy influence the training objective and downstream performance, providing principled criteria for subtokenizer design. When extending the model size to 1.1B parameters, MDM-Prime-v2 demonstrates superior average zero-shot accuracy across eight commonsense reasoning benchmarks, outperforming similar-sized baselines including GPT-Neo, OPT, Pythia, Bloom, SMDM, and TinyLLaMA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。