后训练模型能识别并响应自身生成内容，降低输出不确定性。
From Simulation to Enaction: Post-trained language models recognize and react to their own generations
- 通过后训练让模型意识到自身输出的上下文，从而调节生成行为。
- 自生成输出的熵比外部输入低3到4倍，体现内在一致性。
- 适合研究模型自我意识、生成控制与人类交互的学者参考。
语言模型在预训练阶段是被动预测者，缺乏对自身输出后果的关注。后训练改变了这一点：模型若生成自己的回应，便能从识别自身处于策略内（on-policy）状态中获益。我们发现后训练模型能够识别其策略内生成，并将这种识别隐式编码于输出分布中。具体而言，在不同模型家族和规模下，策略内输出分布熵比策略外低3至4倍。我们发现该现象部分源于模型内部对输入意外性的表征——即根据先前预测判断最近输入词的罕见程度，进而因果性地调节输出熵。例如在开放式提示下，后训练模型会在首个输出词前就缩小对后续话题的不确定性；若用不同主题的填充文本打断此意图，则输出熵升高。我们还测试了模型是否可通过显式语言报告区分策略内与填充上下文，结果表明它们可以，但显式识别依赖不同于隐式识别的机制。
原文摘要 · Abstract (English)
Language models are pretrained as passive predictors with no incentive to model the consequences of their own outputs. Post-training changes this: a model producing its own responses can benefit from recognizing that it is on-policy. We present evidence that post-trained models recognize their on-policy generations, and this recognition is implicitly encoded in their output distributions. In particular, on-policy output distribution entropy is 3--4$\times$ lower than off-policy entropy, across model families and size classes. We trace part of this effect to an internal representation of input surprise, tracking the unlikeliness of the most recent input token according to the model's prior predictions, that causally modulates output entropy. One example of these phenomena can be observed in response to open-ended prompts; post-trained models (unlike pretrained models) collapse their uncertainty over the topic of their upcoming response before the first output token; violating this cached intention with a different-topic prefill results in higher output entropy. We also tested whether models can distinguish on-policy contexts from prefills via explicit verbal report. We find that they can, but that interestingly, this explicit recognition routes through a different mechanism than implicit recognition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。