arXiv:2506.17871cs.CLcs.AI2025-06被引 23

对齐让大模型生成更单一,因输出分布变集中。

LLM Probability Concentration: How Alignment Shrinks the Generative Horizon

  • 用分支因子衡量生成时可选路径数量,越小越确定。
  • 对齐使初始分支因子降低2-5倍,部分达10倍以上。
  • 提示词引导可模拟对齐效果,为可控生成提供新思路。

尽管表现强大,对齐后的大型语言模型常生成缺乏多样性的内容。我们从输出分布的概率集中角度探究此现象。引入‘分支因子’(BF)——一种与分词无关的、衡量生成过程中有效下一步数量的指标。实证分析显示:(1) 随着生成推进,BF通常下降,表明模型逐渐变得更可预测;(2) 对齐调优显著压缩输出分布,使整体分支因子减少2-5倍,初期甚至下降一个数量级(如从12降至1.2)。这一大幅缩减解释了为何对齐模型对解码策略不敏感。基于此,我们发现一致性对复杂推理有意外影响:对齐的思维链(CoT)模型(如DeepSeek-distilled)通过生成更长推理链,进入后期更确定(低BF)阶段,实现更稳定输出。我们推测,对齐并未根本改变模型行为,而是引导其选择特定风格标记(如“Sure”),从而激活原本存在于基础模型中的低熵轨迹。助推实验也证实,用此类标记提示基础模型可同样降低BF。综上,分支因子成为理解与控制大模型输出的重要诊断工具,揭示了对齐如何抑制多样性、思维链如何促进稳定,并展示了如何通过提示引导基础模型偏离多样性路径。

原文摘要 · Abstract (English)

Despite their impressive capabilities, aligned large language models (LLMs) often generate outputs that lack diversity. What drives this consistency in the generation? We investigate this phenomenon through the lens of probability concentration in the model's output distribution. To quantify this concentration, we introduce the *Branching Factor* (BF) -- a token-invariant measure of the effective number of plausible next steps during generation. Our empirical analysis reveals two key findings: (1) BF often decreases as generation progresses, suggesting that LLMs become more predictable as they generate. (2) alignment tuning substantially sharpens the model's output distribution from the outset, reducing BF by a factor of 2-5 overall, and up to an order of magnitude (e.g., from 12 to 1.2) at the beginning positions. This stark reduction helps explain why aligned models often appear less sensitive to decoding strategies. Building on this insight, we find this consistency has surprising implications for complex reasoning. Aligned Chain-of-Thought (CoT) models (e.g., DeepSeek-distilled models), for instance, leverage this effect; by generating longer reasoning chains, they push generation into later, more deterministic (lower BF) stages, resulting in more stable outputs. We hypothesize that alignment tuning does not fundamentally change a model's behavior, but instead steers it toward stylistic tokens (e.g., "Sure") that unlock low-entropy trajectories already present in the base model. This view is supported by nudging experiments, which show prompting base models with such tokens can similarly reduce BF. Together, our findings establish BF as a powerful diagnostic for understanding and controlling LLM outputs - clarifying how alignment reduces variability, how CoT promotes stable generations, and how base models can be steered away from diversity.

大模型生成控制对齐机制概率集中

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。