攻击者伪造模型历史回复，绕过安全机制生成有害内容。
Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Message
- 用伪造的模型历史消息植入恶意指令，再以正常问题触发漏洞。
- 在Gemini-2.0-flash上攻击成功率远超传统用户端越狱方法。
- 揭示对话型AI安全缺陷，适合安全研究者与模型开发者参考。
对话式接口的兴起通过利用对话历史实现复杂推理，显著提升了大模型的可用性。然而，这种依赖也引入了未被探索的攻击面。本文提出一种新型越狱技术——特洛伊木马提示（Trojan Horse Prompting）。攻击者通过伪造提供给API的模型自身历史对话内容，绕过安全机制。将恶意载荷注入模型声称的过往回复中，再搭配一个看似正常的用户提示，即可触发有害内容生成。该漏洞源于不对称安全对齐：模型虽经过大量训练以拒绝用户的有害请求，却缺乏对自身历史对话内容的警惕性。这种对“过去”输出的隐性信任构成了高危漏洞。在Google Gemini-2.0-flash-preview-image-generation上的实验验证表明，该方法的攻击成功率（ASR）显著高于现有用户端越狱手段。研究揭示了现代对话式AI中的根本性安全缺陷，亟需从输入层过滤转向对对话上下文完整性的协议级验证。
原文摘要 · Abstract (English)
The rise of conversational interfaces has greatly enhanced LLM usability by leveraging dialogue history for sophisticated reasoning. However, this reliance introduces an unexplored attack surface. This paper introduces Trojan Horse Prompting, a novel jailbreak technique. Adversaries bypass safety mechanisms by forging the model's own past utterances within the conversational history provided to its API. A malicious payload is injected into a model-attributed message, followed by a benign user prompt to trigger harmful content generation. This vulnerability stems from Asymmetric Safety Alignment: models are extensively trained to refuse harmful user requests but lack comparable skepticism towards their own purported conversational history. This implicit trust in its "past" creates a high-impact vulnerability. Experimental validation on Google's Gemini-2.0-flash-preview-image-generation shows Trojan Horse Prompting achieves a significantly higher Attack Success Rate (ASR) than established user-turn jailbreaking methods. These findings reveal a fundamental flaw in modern conversational AI security, necessitating a paradigm shift from input-level filtering to robust, protocol-level validation of conversational context integrity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。