提出DiffSampling，让模型生成更多样且准确的文本。
DiffSampling: Enhancing Diversity and Accuracy in Neural Text Generation
- 基于概率分布差异截断错误词元，提升生成质量
- 在4个任务中均优于或持平现有方法，采样范围更大
- 适合需要高多样性与准确性的文本生成场景
尽管语言模型能力不断提升，仍频繁复现训练数据内容、产生重复文本，并偏好常见语法模式和词汇。原因可能在于解码策略：主流方法要么仅考虑最可能的词元（降低多样性），要么提高低概率词元可能性（牺牲准确性）。本文提出DiffSampling，通过分析词元概率分布的数学特性，利用连续排序概率间的差异来剔除不恰当词元。此外，还设计了两种变体以修正常见采样策略的细微偏差。在四个不同文本生成任务上的实验表明，该方法在采样更多词元的前提下，质量始终不低于或优于所依赖的现有方法。
原文摘要 · Abstract (English)
Despite their growing capabilities, language models still frequently reproduce content from their training data, generate repetitive text, and favor common grammatical patterns and vocabulary. A possible cause is the decoding strategy: the most common strategies either consider only the most probable tokens, which reduces output diversity, or increase the likelihood of unlikely tokens, compromising output accuracy and correctness. In this paper, we propose DiffSampling, a new decoding method that leverages a mathematical analysis of the token probability distribution to ensure the generation of contextually appropriate text. In particular, the difference between consecutive, sorted probabilities can be used to truncate incorrect tokens. In addition, we also propose two variations of the proposed method that aim to correct the subtle inconsistencies of common sampling strategies. Experiments involving four different text-generation tasks demonstrate that our approach consistently performs at least on par with the existing methods it builds upon in terms of quality, despite sampling from a larger set of tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。