用两个原型标记实现一次前向传播重建长文本，揭示其语义与语法编码机制。
On the Semantic and Syntactic Information Encoded in Proto-Tokens for One-Step Text Reconstruction
- 通过两个原型标记分离语义与句法信息，分析其在重建中的稳定性与注意力模式。
- 关系蒸馏方法可在不牺牲重建质量的前提下，将批次级语义关系注入原型空间。
- 适用于探索非自回归序列生成新范式，尤其适合追求高效推理的场景。
自回归大语言模型逐词生成文本，需进行 n 次前向传播才能生成长度为 n 的序列。近期研究（Mezentsev and Oseledets）表明，冻结的 LLM 可仅用两个学习得到的原型标记，在单次前向传播中重建数百个词，暗示了超越自回归范式的可能。本文研究这些原型标记所编码的信息内容及其在重建与受控约束下的行为。我们设计了一系列实验，旨在解耦两个原型标记中的语义与句法信息，分析 e-token 的稳定性，并可视化重建过程中对 e-token 的注意力模式。最后，我们测试了两种正则化方案以“强制”在 e-token 中引入语义结构：基于锚点的损失与关系蒸馏目标。结果表明，在标准优化下，m-token 比 e-token 更强地捕捉语义信息；基于锚点的约束会显著降低重建准确率；而关系蒸馏可在不牺牲重建质量的前提下，将批次级语义关系迁移至原型标记空间，支持未来非自回归序列到序列系统以原型标记作为中间表示的可行性。
原文摘要 · Abstract (English)
Autoregressive large language models (LLMs) generate text token-by-token, requiring n forward passes to produce a sequence of length n. Recent work, Exploring the Latent Capacity of LLMs for One-Step Text Reconstruction (Mezentsev and Oseledets), shows that frozen LLMs can reconstruct hundreds of tokens from only two learned proto-tokens in a single forward pass, suggesting a path beyond the autoregressive paradigm. In this paper, we study what information these proto-tokens encode and how they behave under reconstruction and controlled constraints. We perform a series of experiments aimed at disentangling semantic and syntactic content in the two proto-tokens, analyzing stability properties of the e-token, and visualizing attention patterns to the e-token during reconstruction. Finally, we test two regularization schemes for "imposing" semantic structure on the e-token using teacher embeddings, including an anchor-based loss and a relational distillation objective. Our results indicate that the m-token tends to capture semantic information more strongly than the e-token under standard optimization; anchor-based constraints trade off sharply with reconstruction accuracy; and relational distillation can transfer batch-level semantic relations into the proto-token space without sacrificing reconstruction quality, supporting the feasibility of future non-autoregressive seq2seq systems that predict proto-tokens as an intermediate representation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。