解释生成模型中行为突变的根源,揭示关键特征在短暂窗口内被锁定的机制。
Blink of an eye: a simple theory for feature localization in generative models
- 用随机定位采样理论统一解释语言与扩散模型的行为突变
- 发现关键特征在生成过程的极短窗口内被确定,且该现象普遍存在于多种模型
- 无需复杂数学工具,适用于广泛模型,对推理失败有预测能力
大语言模型在极短时间内可能发生行为突变,如从编程转为搜索图片;类似现象也出现在推理和越狱攻击中。这种现象不仅限于自回归模型:在扩散模型中,最终输出的关键特征在生成过程的狭窄“临界窗口”内被决定。本文基于随机定位采样框架,提出一个简洁、统一的理论,解释该现象源于生成过程对模型所建模分布的子群体进行局部化。相比现有工作,本理论(1)适用于自回归与扩散模型;(2)不依赖分布假设;(3)在扩散模型上仍能改进已有定量边界;(4)仅使用基础工具,无需随机微积分或统计物理方法。我们还发现其与统计推断中的“全或无”现象存在联系。实证验证表明,临界窗口常与数学与推理基准任务中的解题失败重合。
原文摘要 · Abstract (English)
Large language models can exhibit unexpected behavior in the blink of an eye. In a recent computer use demo, a language model switched from coding to Googling pictures of Yellowstone, and these sudden shifts in behavior have also been observed in reasoning patterns and jailbreaks. This phenomenon is not unique to autoregressive models: in diffusion models, key features of the final output are decided in narrow ``critical windows'' of the generation process. In this work we develop a simple, unifying theory to explain this phenomenon using the formalism of stochastic localization samplers. We show that it emerges generically as the generation process localizes to a sub-population of the distribution it models. While critical windows have been studied at length in diffusion models, existing theory heavily relies on strong distributional assumptions and the particulars of Gaussian diffusion. In contrast to existing work our theory (1) applies to autoregressive and diffusion models; (2) makes no distributional assumptions; (3) quantitatively improves previous bounds even when specialized to diffusions; and (4) requires basic tools and no stochastic calculus or statistical-physics-based machinery. We also identify an intriguing connection to the all-or-nothing phenomenon from statistical inference. Finally, we validate our predictions empirically for LLMs and find that critical windows often coincide with failures in problem solving for various math and reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。