PICO通过分通道隔离用户输入与系统指令，防住提示注入攻击。
PICO: Secure Transformers via Robust Prompt Isolation and Cybersecurity Oversight
- 用双通道分离可信指令与不可信输入，融合前经受门控控制。
- 引入安全专家代理和网络安全知识图谱，提升对抗攻击的推理能力。
- 适合需要高安全性的AI系统部署，如金融、医疗对话场景。
我们提出一种鲁棒的Transformer架构PICO（Prompt Isolation and Cybersecurity Oversight），以防范提示注入攻击并确保响应生成的安全可靠。该框架通过双通道结构将可信系统指令与不可信用户输入在内部处理中严格分离，仅在受控的门控融合机制下合并输出。同时，在Mixture-of-Experts（MoE）框架中集成专用安全专家代理，并引入网络安全知识图谱（CKG）提供领域特定推理支持。训练设计保证系统提示分支不可更改，其余网络则学习安全应对外部恶意输入。该框架以通用数学形式呈现，结合Transformer具体实现，并通过政策傀儡攻击等假设案例进行阐释。最优效果可通过从头训练实现，但我们也提出了低成本的微调方案。
原文摘要 · Abstract (English)
We propose a robust transformer architecture designed to prevent prompt injection attacks and ensure secure, reliable response generation. Our PICO (Prompt Isolation and Cybersecurity Oversight) framework structurally separates trusted system instructions from untrusted user inputs through dual channels that are processed independently and merged only by a controlled, gated fusion mechanism. In addition, we integrate a specialized Security Expert Agent within a Mixture-of-Experts (MoE) framework and incorporate a Cybersecurity Knowledge Graph (CKG) to supply domain-specific reasoning. Our training design further ensures that the system prompt branch remains immutable while the rest of the network learns to handle adversarial inputs safely. This PICO framework is presented via a general mathematical formulation, then elaborated in terms of the specifics of transformer architecture, and fleshed out via hypothetical case studies including Policy Puppetry attacks. While the most effective implementation may involve training transformers in a PICO-based way from scratch, we also present a cost-effective fine-tuning approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。