arXiv:2607.29378cs.CLcs.LG2026-07

无需模型权重,通过反向预测前一词实现精准提示还原。

PTP: Previous-Token Prediction based LLM Inversion for Near-Exact Prompt Reconstruction

论文配图:PTP: Previous-Token Prediction based LLM Inversion for Near-Exact Prompt Reconstruction
图 1 · 摘自论文原文
  • 用目标模型自动生成数据,训练反向语言模型进行前词预测。
  • 在多个数据集和模型上均优于已有方法,提示重建准确率显著提升。
  • 适合隐私安全研究者,可用于逆向分析大模型输出的原始输入。

大型语言模型(LLM)通过自回归方式逐词生成文本,导致提示与输出之间存在多对多映射关系,使从输出反推提示变得困难。以往工作将提示恢复视为语义重构任务,依赖在外部大规模数据上微调预训练序列到序列模型,并需访问模型权重或输出概率。本文提出一种无需辅助工具的黑箱逆向方法:完全从零训练一个显式逆向语言模型,其训练数据由目标模型自身生成。该逆向模型采用前词预测机制,与正向的下一词预测形成对称生成关系,从而实现高保真提示重建。此外,通过采样可获得多种诱导相同输出的提示,具备良好泛化性和跨模型迁移能力。在基于词的提示与输出重建评估指标上,本方法全面超越现有工作。

原文摘要 · Abstract (English)

Large language models (LLMs) generate text by auto-regressively sampling the next token. This inherently leads to a many-to-many mapping between prompts and responses, complicating the task of inferring prompts from observed outputs. Prior work on LLM inversion frames prompt recovery as a semantic reconstruction task. They rely on fine-tuning pretrained sequence-to-sequence models on large external datasets--and requiring access to model weights or logits--to generate semantically plausible prompts. In contrast, we present a functional approach to inverting a given LLM in a black-box setting, without auxiliary aids. We train an explicit inverse language model entirely from scratch on data synthetically generated from the target LLM itself. Analogous to forward next-token prediction, our inverse model is trained using previous-token prediction, establishing a generative link between the forward and inverse processes that enables faithful prompt reconstruction. Moreover, it naturally supports diverse prompt reconstructions through sampling, whereby all such prompts induce similar responses under the forward, target LLM. Our approach generalises across datasets and exhibits transferability in reconstructing prompts from responses generated by different LLMs. Further, across the set of token based evaluation metrics for prompt and response reconstructions, our approach outperforms prior work.

LLM逆向提示还原黑箱逆向生成建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。