从LLM单个输出向量还原原始文本，实现半数词汇的精准重建。
Rep2Text: Decoding Full Text from a Single LLM Token Representation
- 用可训练适配器将模型输出映射到解码器嵌入空间，实现自回归文本重建。
- 16词序列中平均恢复约一半词汇，语义连贯性保持良好。
- 对临床等分布外数据有强泛化能力，适合研究模型可逆性的学者。
大型语言模型（LLMs）在各类任务中取得显著进展，但其内部机制仍不透明。本文探讨一个基本问题：能否从LLM的最后一个输出向量中恢复原始输入文本？为此，我们提出Rep2Text框架，通过可训练适配器将目标模型的最后输出表示映射到解码语言模型的词嵌入空间，并自回归重建输入文本。在多种模型组合（Llama-3.1-8B、Gemma-7B、Mistral-7B-v0.1、Llama-3.2-3B等）上的实验表明，平均而言，在16词序列中约一半的词可被成功恢复，且语义连贯性良好。进一步分析发现存在明显的信息瓶颈效应：随着序列长度增加，词级恢复率下降，但语义信息仍相对保留。此外，反转任务中的缩放效应不显著。框架还展现出对分布外临床数据的稳健泛化能力。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved remarkable progress across diverse tasks, yet their internal mechanisms remain largely opaque. In this work, we investigate a fundamental question: to what extent can the original input text be recovered from a single last-token representation in an LLM? To this end, we propose Rep2Text, a novel framework for decoding text from last-token representations. Rep2Text employs a trainable adapter that maps a target model's last-token representation into the token embedding space of a decoding language model, which then autoregressively reconstructs the input text. Experiments across various model combinations (Llama-3.1-8B, Gemma-7B, Mistral-7B-v0.1, Llama-3.2-3B, etc.) show that, on average, roughly half of the tokens in 16-token sequences can be recovered from this compressed representation while preserving strong semantic coherence. Further analysis reveals a clear information bottleneck effect: as sequence length increases, token-level recovery declines, while semantic information remains relatively well preserved. We also find that scaling effects are less pronounced in inversion tasks. Finally, our framework demonstrates robust generalization to out-of-distribution clinical data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。