用几何距离优化语言模型采样,提升生成质量和创意。
Geometry-Aware Decoding with Wasserstein-Regularized Truncation and Mass Penalties for Large Language Models
- 基于词元嵌入几何设计截断规则,平衡概率与熵。
- 在四个数据集上最高提升33.7%准确率,且增强创造力。
- 兼容现有解码接口,可直接用于主流模型。
大型语言模型在开放生成中需权衡多样性与逻辑一致性。现有基于截断的采样方法多为启发式,主要依赖概率质量与熵,忽略词元空间的语义几何结构。本文提出Top-W,一种基于Wasserstein距离定义的几何感知截断规则,使裁剪后分布贴近原分布,同时显式平衡保留的概率质量与保留集合的熵。理论推导出固定势能子集更新的闭式结构:根据质量-熵权衡,最优裁剪要么退化为单个词元,要么呈一维前缀形式,可通过线性扫描高效求解。我们采用高效的几何势函数(最近集或k-NN)实现Top-W,并结合交替解码流程,保持标准截断-采样接口不变。在GSM8K、GPQA、AlpacaEval和MT-Bench四个基准上,对三种指令微调模型的实验表明,Top-W持续优于现有最先进解码方法,最高提升达33.7%。此外,发现Top-W不仅提升准确性,还增强判别式开放生成评估下的创造力。
原文摘要 · Abstract (English)
Large language models (LLMs) must balance diversity and creativity against logical coherence in open-ended generation. Existing truncation-based samplers are effective but largely heuristic, relying mainly on probability mass and entropy while ignoring semantic geometry of the token space. We present Top-W, a geometry-aware truncation rule that uses Wasserstein distance-defined over token-embedding geometry-to keep the cropped distribution close to the original, while explicitly balancing retained probability mass against the entropy of the kept set. Our theory yields a simple closed-form structure for the fixed-potential subset update: depending on the mass-entropy trade-off, the optimal crop either collapses to a single token or takes the form of a one-dimensional prefix that can be found efficiently with a linear scan. We implement Top-W using efficient geometry-based potentials (nearest-set or k-NN) and pair it with an alternating decoding routine that keeps the standard truncation-and-sampling interface unchanged. Extensive experiments on four benchmarks (GSM8K, GPQA, AlpacaEval, and MT-Bench) across three instruction-tuned models show that Top-W consistently outperforms prior state-of-the-art decoding approaches achieving up to 33.7% improvement. Moreover, we find that Top-W not only improves accuracy-focused performance, but also boosts creativity under judge-based open-ended evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。