仅用两个学习嵌入，模型单步生成数百词，突破传统自回归限制。
Exploring the Hidden Capacity of LLMs for One-Step Text Generation
- 用两个可学习嵌入，单次并行前向传播生成数百词。
- 冻结的LLM在不自回归情况下也能准确生成文本，验证了隐含能力。
- 适合想高效生成长文本且不想重训练的开发者使用。
最近研究发现，大语言模型（LLMs）可通过仅一个训练过的输入嵌入，自回归重建长达数千个标记的文本。本文探索自回归解码是否为该重建所必需。我们发现，当提供两个学习嵌入时，冻结的LLM可在一次并行前向传播中生成数百个准确的标记，揭示了自回归LLM中未被充分探索的多标记生成潜力。我们分析了这些嵌入并刻画其编码信息。实证表明，尽管这些表示对给定文本不唯一，但在嵌入空间中形成连通且局部的区域，暗示可训练实用编码器。此类表示的存在提示：通过学习输入编码器，可直接在现成的LLM上实现原生多标记生成，无需大量重训练，有望克服自回归解码的根本瓶颈，同时复用已训练模型。
原文摘要 · Abstract (English)
A recent study showed that large language models (LLMs) can reconstruct surprisingly long texts - up to thousands of tokens - via autoregressive generation from just one trained input embedding. In this work, we explore whether autoregressive decoding is essential for such reconstruction. We show that frozen LLMs can generate hundreds of accurate tokens in just one token-parallel forward pass, when provided with only two learned embeddings. This reveals a surprising and underexplored multi-token generation capability of autoregressive LLMs. We examine these embeddings and characterize the information they encode. We also empirically show that, although these representations are not unique for a given text, they form connected and local regions in embedding space - suggesting the potential to train a practical encoder. The existence of such representations hints that multi-token generation may be natively accessible in off-the-shelf LLMs via a learned input encoder, eliminating heavy retraining and helping to overcome the fundamental bottleneck of autoregressive decoding while reusing already-trained models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。