用快速代理指标预测大模型对抗攻击鲁棒性，节省大量计算成本
Fast Proxies for LLM Robustness Evaluation
- 设计快速代理指标，通过嵌入空间攻击和直接提示预测真实攻击效果
- 与完整攻击集合相关性高达0.94（Spearman），计算成本降低千倍
- 适合需要高效评估模型安全性的研究者和工程师使用
评估大模型对对抗攻击的鲁棒性对于安全部署至关重要，但当前红队测试方法往往成本过高。本文比较了快速代理指标预测大模型在模拟攻击集合下真实鲁棒性的能力，从而无需实际运行高成本攻击即可估算模型鲁棒性。具体考虑基于梯度下降的嵌入空间攻击、预填充攻击和直接提示攻击。尽管直接提示攻击本身未达高攻击成功率（ASR），但其与嵌入空间攻击均能有效预测攻击成功概率，与完整攻击集合的相关系数分别为 $r_p=0.87$（线性）和 $r_s=0.94$（Spearman秩相关），同时将计算成本降低三个数量级。
原文摘要 · Abstract (English)
Evaluating the robustness of LLMs to adversarial attacks is crucial for safe deployment, yet current red-teaming methods are often prohibitively expensive. We compare the ability of fast proxy metrics to predict the real-world robustness of an LLM against a simulated attacker ensemble. This allows us to estimate a model's robustness to computationally expensive attacks without requiring runs of the attacks themselves. Specifically, we consider gradient-descent-based embedding-space attacks, prefilling attacks, and direct prompting. Even though direct prompting in particular does not achieve high ASR, we find that it and embedding-space attacks can predict attack success rates well, achieving $r_p=0.87$ (linear) and $r_s=0.94$ (Spearman rank) correlations with the full attack ensemble while reducing computational cost by three orders of magnitude.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。