arXiv:2505.13348cs.CL2025-05被引 14

发现大模型评分系统易受恶意提示攻击,可轻松篡改判断结果。

Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks

  • 用优化方法生成恶意后缀,干扰模型评分决策。
  • 攻击成功率超30%,能有效改变模型推理理由。
  • 适合关注AI评估安全性的研究人员和开发者。

大型语言模型(LLMs)正被广泛用于评估机器生成文本的质量,相比人工标注更具可扩展性和成本优势。然而,这类系统在对抗性操纵下的可靠性和安全性仍存疑。本文研究了LLM-as-a-Judge架构对提示注入攻击的脆弱性,提出两种攻击策略:直接攻击最终判决结果的对比削弱攻击(CUA),以及操纵模型推理过程的合理性攻击(JMA)。采用贪心坐标梯度(GCG)方法,在待比较的响应中添加恶意后缀。在MT-Bench人类判断数据集上,使用Qwen2.5-3B-Instruct与Falcon3-3B-Instruct两个开源指令微调模型进行实验,结果显示显著脆弱性:CUA攻击成功率(ASR)超过30%,JMA也表现出明显效果。这些发现揭示了当前基于LLM的评估系统存在重大安全隐患,亟需构建鲁棒防御机制,并推动对抗性评估与可信性研究。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are increasingly employed as evaluators (LLM-as-a-Judge) for assessing the quality of machine-generated text. This paradigm offers scalability and cost-effectiveness compared to human annotation. However, the reliability and security of such systems, particularly their robustness against adversarial manipulations, remain critical concerns. This paper investigates the vulnerability of LLM-as-a-Judge architectures to prompt-injection attacks, where malicious inputs are designed to compromise the judge's decision-making process. We formalize two primary attack strategies: Comparative Undermining Attack (CUA), which directly targets the final decision output, and Justification Manipulation Attack (JMA), which aims to alter the model's generated reasoning. Using the Greedy Coordinate Gradient (GCG) optimization method, we craft adversarial suffixes appended to one of the responses being compared. Experiments conducted on the MT-Bench Human Judgments dataset with open-source instruction-tuned LLMs (Qwen2.5-3B-Instruct and Falcon3-3B-Instruct) demonstrate significant susceptibility. The CUA achieves an Attack Success Rate (ASR) exceeding 30\%, while JMA also shows notable effectiveness. These findings highlight substantial vulnerabilities in current LLM-as-a-Judge systems, underscoring the need for robust defense mechanisms and further research into adversarial evaluation and trustworthiness in LLM-based assessment frameworks.

大模型评估提示攻击安全性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。