用分层扩散模型实现语义粒度渐进预测，提升文本生成质量。
Next Semantic Scale Prediction via Hierarchical Diffusion Language Models
- 通过分层词表将细粒度语义逐步映射到粗粒度表示
- 在多个数据集上生成困惑度显著低于基线模型
- 适合追求高质量文本生成的NLP研究者
本文提出分层扩散语言模型（HDLM），一种用于语言建模的新型离散扩散模型。HDLM基于分层词表结构，将具有细致语义的低层级词元，以满射方式映射到语义更粗略的高层级词元。前向过程中，每个词元根据调度器独立扰动为更高层级的祖先词元；反向过程中，模型逐步预测下一个更详细的语义。整体上，HDLM提供了一个通用的、随时间变化的下一级语义尺度预测过程。我们推导出扩散证据下界（ELBO）的闭式表达，并证明HDLM可灵活实现，且包含现有MDLM作为特例。我们还提出了基于洞察的实用训练技巧。大量文本生成实验验证了其有效性，结果显示其在验证和生成困惑度上持续优于基线模型。
原文摘要 · Abstract (English)
In this paper we introduce Hierarchical Diffusion Language Models (HDLM) -- a novel family of discrete diffusion models for language modeling. HDLM builds on a hierarchical vocabulary where low-level tokens with detailed semantics are surjectively mapped to high-level tokens with coarse-grained meanings. In the forward process, each token is independently perturbed to its higher-level ancestor with more abstract semantics according to the scheduler, while in the reverse process the model progressively predicts the next, more detailed semantics. Taken together, HDLM provides a general time-varying next semantic scale prediction process for language modeling. We derive closed-form expressions for the diffusion Evidence Lower Bound (ELBO), and show that HDLM can be implemented in a flexible manner while including the existing MDLM as a special case. We also propose practical training techniques based on the insights. Extensive text generation experiments validate the effectiveness of HDLM, which demonstrates consistently lower validation and generative perplexity than baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。