对比局部与全局归一化，发现局部方法虽扭曲分布却更有效。
Local and Global Decoding in Text Generation
- 提出全局归一化版本的top-k和top-π解码方法
- 实验显示全局方法在多数配置下表现更差
- 揭示局部畸变可能是解码有效性的关键因素
文本生成依赖于从语言模型分布中采样字符串的解码算法。传统方法如top-k和top-π采用局部归一化，会扭曲输出分布。本文研究该扭曲的影响，引入这些方法的全局归一化版本，并提出一种独立的Metropolis-Hastings算法，无需显式计算即可近似全局归一化分布的采样。通过Pythia语言模型，在多种超参数下对比局部与全局归一化性能。结果表明，在大多数配置中，全局解码表现劣于对应局部解码方法——尽管其保持了分布完整性。这说明局部畸变对解码算法的有效性具有重要意义。
原文摘要 · Abstract (English)
Text generation, a key component in applications such as dialogue systems, relies on decoding algorithms that sample strings from a language model distribution. Traditional methods, such as top-$k$ and top-$π$, apply local normalisation to the model's output distribution, which can distort it. In this paper, we investigate the effect of this distortion by introducing globally-normalised versions of these decoding methods. Additionally, we propose an independent Metropolis-Hastings algorithm to approximate sampling from globally-normalised distributions without explicitly computing them. Our empirical analysis compares the performance of local and global normalisation across two decoding algorithms (top-$k$ and top-$π$) with various hyperparameters, using Pythia language models. Results show that, in most configurations, global decoding performs worse than the local decoding version of the same algorithms -- despite preserving the distribution's integrity. Our results suggest that distortion is an important feature of local decoding algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。