发现自回归模型本质是能量模型，揭示其隐含的前瞻能力
Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction
- 从概率链式法则出发，建立自回归与能量模型的函数空间一一对应
- 证明两者在监督学习中等价，且可理论保证蒸馏误差
- 为理解文本生成中的前瞻机制提供新视角，适合模型原理研究者
自回归模型（ARMs）目前是大语言模型（LLMs）的主流范式。能量基模型（EBMs）是另一类模型，虽在LLM发展中较少使用，但天然适合后训练对齐中的最优策略建模。本文提供两类模型的统一视角：以概率链式法则为起点，建立ARMs与EBMs在函数空间的显式双射关系，该关系对应最大熵强化学习中软贝尔曼方程的特例。基于此双射，我们推导出ARMs与EBMs在监督学习中的等价性，并进一步给出EBMs蒸馏为ARMs的理论误差界。结果揭示了尽管基于逐词预测，自回归模型仍具备前瞻规划能力。
原文摘要 · Abstract (English)
Autoregressive models (ARMs) currently constitute the dominant paradigm for large language models (LLMs). Energy-based models (EBMs) represent another class of models, which have historically been less prevalent in LLM development, yet naturally characterize the optimal policy in post-training alignment. In this paper, we provide a unified view of these two model classes. Taking the chain rule of probability as a starting point, we establish an explicit bijection between ARMs and EBMs in function space, which we show to correspond to a special case of the soft Bellman equation in maximum entropy reinforcement learning. Building upon this bijection, we derive the equivalence between supervised learning of ARMs and EBMs. Furthermore, we analyze the distillation of EBMs into ARMs by providing theoretical error bounds. Our results provide insights into the ability of ARMs to plan ahead, despite being based on the next-token prediction paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。