提出动态调整生成概率带宽的方法，让语言模型自适应调节创造与推理的平衡。
Top-b: Entropic Regulation of Relative Probability Bands in Autoregressive Language Processes
- 基于实时熵值动态调节候选词范围，取代固定阈值策略。
- 在GPQA和GSM8K上降低生成熵与解码方差，保持推理准确率。
- 适合需要稳定高质输出的复杂推理或创造性任务场景。
概率语言生成器理论上可视为离散随机过程，但标准解码策略（如Top-k、Top-p）采用静态截断规则，难以适应自然语言中动态变化的信息密度。这种不匹配常导致次优权衡：静态边界对高熵创作生成过于严格，对低熵逻辑推理又过于宽松。本文将生成过程建模为相对概率流形上的轨迹，提出Top-b（自适应相对带采样）解码策略，通过与模型分布瞬时香农熵严格耦合的动态带宽系数调节候选集。我们提供理论框架，证明Top-b是尾部分布的方差最小化算子。在GPQA和GSM8K基准上的实证验证表明，Top-b显著降低生成熵与解码间方差，同时保持有竞争力的推理准确率，有效逼近自调节控制机制。
原文摘要 · Abstract (English)
Probabilistic language generators are theoretically modeled as discrete stochastic processes, yet standard decoding strategies (Top-k, Top-p) impose static truncation rules that fail to accommodate the dynamic information density of natural language. This misalignment often forces a suboptimal trade-off: static bounds are either too restrictive for high-entropy creative generation or too permissive for low-entropy logical reasoning. In this work, we formalize the generation process as a trajectory through a relative probability manifold. We introduce Top-b (Adaptive Relative Band Sampling), a decoding strategy that regulates the candidate set via a dynamic bandwidth coefficient coupled strictly to the instantaneous Shannon entropy of the model's distribution. We provide a theoretical framework demonstrating that Top-b acts as a variance-minimizing operator on the tail distribution. Empirical validation on GPQA and GSM8K benchmarks indicates that Top-b significantly reduces generation entropy and inter-decoding variance while maintaining competitive reasoning accuracy, effectively approximating a self-regulating control system for autoregressive generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。