arXiv:2509.02510cs.CLcs.AI2025-09NeurIPS被引 12

提出Top-H解码,让生成文本更创意且连贯。

Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation

  • 基于熵约束优化,用贪心算法平衡创意与逻辑。
  • 在创意写作上比最优方法提升25.63%。
  • 适合需要高创意又不失连贯性的写作场景。

大语言模型在开放式文本生成中常面临创意与连贯性难以兼顾的挑战。现有截断采样方法如温度调节、top-p(核)采样和min-p采样虽试图调和此矛盾,但未能有效利用模型置信度信息。本文提出Top-H解码,从理论出发建立熵约束最小偏离问题,并证明其等价于难解的熵约束质量最大化(ECMM)问题。进而设计出一种计算高效的贪心算法求解。大量实验证明,Top-H在创意写作基准上相较当前最优的min-p采样最高提升25.63%,同时在GPQA、GSM8K和MT-Bench等问答数据集上保持鲁棒性。通过‘大模型作为裁判’评估,证实其在高温度下仍能生成连贯内容。代码已开源。

原文摘要 · Abstract (English)

Large language models (LLMs), despite their impressive performance across a wide range of tasks, often struggle to balance two competing objectives in open-ended text generation: fostering diversity and creativity while preserving logical coherence. Existing truncated sampling techniques, including temperature scaling, top-\$p\$ (nucleus) sampling, and min-\$p\$ sampling, aim to manage this trade-off. However, they exhibit limitations, particularly in the effective incorporation of the confidence of the model into the corresponding sampling strategy. For example, min-\$p\$ sampling relies on a single top token as a heuristic for confidence, eventually underutilizing the information of the probability distribution. Toward effective incorporation of the confidence of the model, in this paper, we present **top-H** decoding. We first establish the theoretical foundation of the interplay between creativity and coherence in truncated sampling by formulating an **entropy-constrained minimum divergence** problem. We then prove this minimization problem to be equivalent to an **entropy-constrained mass maximization** (ECMM) problem, which is NP-hard. Finally, we present top-H decoding, a computationally efficient greedy algorithm to solve the ECMM problem. Extensive empirical evaluations demonstrate that top-H outperforms the state-of-the-art (SoTA) alternative of min-\$p\$ sampling by up to **25.63%** on creative writing benchmarks, while maintaining robustness on question-answering datasets such as GPQA, GSM8K, and MT-Bench. Additionally, an *LLM-as-judge* evaluation confirms that top-H indeed produces coherent outputs even at higher temperatures, where creativity is especially critical. In summary, top-H advances SoTA in open-ended text generation and can be *easily integrated* into creative writing applications. The code is available at https://github.com/ErfanBaghaei/Top-H-Decoding.

文本生成解码策略大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。