通过污染对齐数据，让大模型更容易被提示注入攻击。
Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignment
- 用恶意样本污染对齐数据，影响模型对齐过程。
- 仅少量污染数据就使模型对多种注入攻击更易受攻击。
- 攻击隐蔽性强，不影响常规评测表现，适合隐蔽渗透。
提示注入攻击通过在原始提示中插入恶意指令，诱导大语言模型执行攻击者指定的任务，构成重大安全威胁。现有攻击多聚焦于推理时构造注入提示，将模型视为静态目标。实验表明这些攻击虽有一定成功率，但仍有提升空间。本文提出更基础的攻击方式：通过污染模型对齐过程，增强未来提示注入攻击的效果。具体提出PoisonedAlign方法，通过生成特定恶意对齐样本污染对齐数据集。在五个大模型和两个对齐数据集上的实验显示，即使少量对齐数据被污染，模型对多种提示注入攻击的脆弱性显著提升。关键的是，该漏洞在标准能力基准测试中几乎不体现，使自动化检测难以发现。攻击代码已开源：https://github.com/Sadcardation/PoisonedAlign。
原文摘要 · Abstract (English)
Prompt injection attack, where an attacker injects a prompt into the original one, aiming to make an Large Language Model (LLM) follow the injected prompt to perform an attacker-chosen task, represent a critical security threat. Existing attacks primarily focus on crafting these injections at inference time, treating the LLM itself as a static target. Our experiments show that these attacks achieve some success, but there is still significant room for improvement. In this work, we introduces a more foundational attack vector: poisoning the LLM's alignment process to amplify the success of future prompt injection attacks. Specifically, we propose PoisonedAlign, a method that strategically creates poisoned alignment samples to poison an LLM's alignment dataset. Our experiments across five LLMs and two alignment datasets show that when even a small fraction of the alignment data is poisoned, the resulting model becomes substantially more vulnerable to a wide range of prompt injection attacks. Crucially, this vulnerability is instilled while the LLM's performance on standard capability benchmarks remains largely unchanged, making the manipulation difficult to detect through automated, general-purpose performance evaluations. The code for implementing the attack is available at https://github.com/Sadcardation/PoisonedAlign.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。