研究大模型生成时对未选路径的感知能力
Are language models aware of the road not taken? Token-level uncertainty and hidden state dynamics
- 用隐藏层激活值控制与预测推理中的不确定性
- 激活干预在未确定答案时效果最好，相关性显著
- 模型隐式存储多种可能路径，适合研究决策机制
语言模型生成文本时，单个词元的选择可能导致截然不同的推理路径，使不确定性难以量化。本文探究推理型语言模型是否在生成过程中表征了可选的其他路径。通过分析隐藏激活值，我们发现模型在不同词元处的不确定性与其被激活干预的难易程度存在明显相关性。这表明激活干预在模型尚未确定最终答案时最为有效。此外，隐藏激活还能预测模型未来的输出分布，证明模型隐式存储了可能路径的空间。
原文摘要 · Abstract (English)
When a language model generates text, the selection of individual tokens might lead it down very different reasoning paths, making uncertainty difficult to quantify. In this work, we consider whether reasoning language models represent the alternate paths that they could take during generation. To test this hypothesis, we use hidden activations to control and predict a language model's uncertainty during chain-of-thought reasoning. In our experiments, we find a clear correlation between how uncertain a model is at different tokens, and how easily the model can be steered by controlling its activations. This suggests that activation interventions are most effective when there are alternate paths available to the model -- in other words, when it has not yet committed to a particular final answer. We also find that hidden activations can predict a model's future outcome distribution, demonstrating that models implicitly represent the space of possible paths.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。