用能量模型思路改造GPT，让生成过程像在能量场中寻优。
NRGPT: An Energy-based Alternative for GPT
- 将GPT的生成视为能量场中的动态探索，引入梯度下降机制。
- 在莎士比亚文本、ListOPS任务和OpenWebText上表现良好。
- 训练极长时才轻微过拟合，可能更抗过拟合，适合长期训练场景。
生成式预训练变换器（GPT）是当前语言建模最流行的架构。能量基模型（EBM）是一种不同范式，将推理视为在能量景观上的动态过程。我们提出对GPT进行最小修改，使其与EBM框架统一。我们的模型——eNeRgy-GPT（NRGPT），将推理步骤概念化为在能量景观上探索词元的过程。我们证明并实证验证，在特定条件下，该探索等价于梯度下降，尽管这不必然带来最优性能。实验表明，该模型在简单语言（Shakespeare数据集）、代数类ListOPS任务以及开放网络文本（OpenWebText）语言建模等丰富场景中表现良好。此外，观察发现模型仅在极长时间训练后才出现过拟合，可能具有更强的抗过拟合能力。
原文摘要 · Abstract (English)
Generative Pre-trained Transformer (GPT) architectures are the most popular design for language modeling. Energy-based modeling is a different paradigm that views inference as a dynamical process operating on an energy landscape. We propose a minimal modification of the GPT setting to unify it with the EBM framework. The inference step of our model, which we call eNeRgy-GPT (NRGPT), is conceptualized as an exploration of the tokens on the energy landscape. We prove, and verify empirically, that under certain circumstances this exploration becomes gradient descent, although they don't necessarily lead to the best performing models. We demonstrate that our model performs well for simple language (Shakespeare dataset), algebraic ListOPS tasks, and richer settings such as OpenWebText language modeling. We also observe that our models may be more resistant to overfitting, doing so only during very long training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。