发现大模型提示优化存在安全漏洞，恶意反馈可大幅降低提示质量。
Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers
- 通过伪造评分反馈实施攻击，无需访问奖励模型。
- 攻击成功率最高提升48%（ΔASR=0.48），远超查询污染攻击。
- 提出轻量级防御方法，将攻击影响降至0.07，且不损害功能。
大型语言模型（LLM）系统正广泛用于聊天机器人、计算机助手和自主机器人等日常AI应用，其性能常依赖精心设计的提示词。基于LLM的提示优化器通过评分反馈迭代改进提示，但该优化阶段的安全性尚未受到充分关注。本文首次系统分析了基于LLM的提示优化中的投毒风险。利用HarmBench测试发现，系统对被操纵的反馈攻击比对查询投毒更脆弱：反馈攻击可使攻击成功率（ASR）最高提升ΔASR = 0.48。我们提出一种无需访问奖励模型的简单虚假奖励攻击，显著增加系统脆弱性。同时，提出一种轻量级高亮防御机制，可将虚假奖励导致的ΔASR从0.23降至0.07，且不降低任务实用性。结果表明，提示优化流程是首个需重点防护的攻击面，亟需加强反馈通道与优化框架的安全保障。
原文摘要 · Abstract (English)
Large language model (LLM) systems increasingly power everyday AI applications such as chatbots, computer-use assistants, and autonomous robots, where performance often depends on manually well-crafted prompts. LLM-based prompt optimizers reduce that effort by iteratively refining prompts from scored feedback, yet the security of this optimization stage remains underexamined. We present the first systematic analysis of poisoning risks in LLM-based prompt optimization. Using HarmBench, we find systems are substantially more vulnerable to manipulated feedback than to query poisoning alone: feedback-based attacks raise attack success rate (ASR) by up to ΔASR = 0.48. We introduce a simple fake reward attack that requires no access to the reward model and significantly increases vulnerability. We also propose a lightweight highlighting defense that reduces the fake reward ΔASR from 0.23 to 0.07 without degrading utility. These results establish prompt optimization pipelines as a first-class attack surface and motivate stronger safeguards for feedback channels and optimization frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。