共享嵌入架构下指令与数据无法分离,导致提示注入攻击无法根治。
On the Inseparability of Instructions and Data in Shared-Embedding Sequence Models
- 提出语义忠实控制(SFC)概念,要求控制行为仅依赖输入语义
- 证明在共享管道中无法实现SFC,存在三重数学不可能性
- 建议从架构上分离指令与数据通道,而非依赖现有防御机制
提示注入是集成大模型应用的首要安全风险,但迄今所有防御方案均被攻破。我们证明这并非偶然:在缺乏强制指令与数据分离的共享嵌入架构中,完全防止提示注入在数学上是不可能的。我们将提示系统形式化为包含控制性动作(如拒绝、授权、策略路由、内存写入)的提示动作模型,并定义了语义忠实控制(SFC)——即控制行为仅取决于未受信输入的语义,而非其编码方式。我们通过三个结果证明:溯源恢复不可能(共享表示使可信与不可信内容在统计上无法区分,受限于总变差距离);控制路径暴露(不可信令牌通过决定输出的相同注意力值聚合进入控制计算);有限覆盖不变性间隙(有限训练无法对无限语义等价类保证不变性)。各量均基于生产级分词器和模型的实测数据。该问题为结构性缺陷,非当前防御之不足。其本质类似于冯·诺依曼机中的代码-数据混淆,曾引发缓冲区溢出,需历经多层防御(DEP、Write-XOR-Execute、ASLR、栈警报、内存安全语言)数十年才得以抑制。结论相同:提示注入无法仅靠更优的管道内分类或对齐消除,必须进行指令与数据通道的架构分离。
原文摘要 · Abstract (English)
Prompt injection is the top security risk for LLM-integrated applications, yet every defense proposed so far has been broken. We prove this is not a coincidence: in shared-embedding architectures that lack enforced control-data separation, perfect prompt-injection prevention is mathematically impossible. We formalize prompted systems as Prompted Action Models whose outputs include control-authoritative actions: refusal decisions, tool authorization, policy routing, and memory writes. We define Semantic-Faithful Control (SFC), the property that such behavior depends only on the meaning of untrusted input, not on how it is encoded. We then prove SFC is unachievable within the shared pipeline, via three results: a provenance-recovery impossibility (shared representations make trusted and untrusted content statistically inseparable, bounded by total variation distance); control-path exposure (untrusted tokens enter control-relevant computation through the same attention value-aggregation that determines outputs); and a finite-coverage invariance gap (finite training cannot certify invariance over infinite semantic-equivalence classes). We ground each quantity in measurements on production tokenizers and models. The result is structural, not a gap in current defenses. It mirrors the code-data confusion in Von Neumann machines that gives rise to buffer overflows, a vulnerability class that took decades of layered defenses (DEP, Write-XOR-Execute, ASLR, stack canaries, and ultimately memory-safe languages) to contain, because no single mechanism sufficed. The implication is the same: prompt injection cannot be eliminated by better in-pipeline classification or alignment alone. It requires architectural separation of instruction and data channels. We identify the root cause and the class of solution it demands.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。