现有评测高估了恶意提示检测模型泛化能力，新方法揭示数据分布偏移下的真实性能差距。
When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift
- 用隐藏层激活训练分类器，采用留一数据集外评估法检验真实泛化能力
- 标准交叉验证的AUC比留一法高出8.0-16.5点，单个数据集准确率差距达1-25点
- 发现44%关键特征是数据集专属捷径，适合关注安全评估可靠性的研究者
检测提示注入、越狱攻击和有害请求对安全部署大语言模型代理至关重要，但当前评估方法过度高估模型泛化能力。我们在18个数据集（含恶意提示与正常来源）上训练基于激活的分类器（线性探测器），提出留一数据集外（LODO）评估策略——被留出的数据集在训练中从未出现。在三个模型家族的四款模型（Llama-3.1-8B、Gemma-3-27B、Qwen-3.5-2B/4B）上，标准交叉验证的综合AUC比LODO高8.0-16.5点，单数据集测试与LODO的准确率差距为1-25点。分析表明，线性探测器的特征系数在LODO下稳定性差，通过稀疏自编码器（SAE）特征定义保留度指标，识别出28-44%的顶级特征为数据集依赖捷径；仅用数据集标识就可达96.6%准确率，且数据集识别与安全相关子空间部分重叠。对抗训练、子空间投影、样本重加权和类别平衡等主流领域泛化改进方法均无法弥合差距。最后，我们证明加权的SAE归因能过滤数据集伪影，提供更可靠的逐提示解释。代码已开源：https://github.com/maxf-zn/prompt-mining
原文摘要 · Abstract (English)
Detecting prompt injection, jailbreak attacks, and harmful requests is critical for deploying LLM-based agents safely, yet current evaluation practices in this literature overestimate generalization. We train activation-based classifiers (linear probes on LLM hidden states) on a benchmark of 18 datasets (prompt attacks plus benign sources) and propose Leave-One-Dataset-Out (LODO) evaluation, where the held-out dataset is never seen during training. Across four LLMs from three families (Llama-3.1-8B, Gemma-3-27B, Qwen-3.5-2B/4B), standard cross-validation reports a pooled AUC 8.0-16.5 points higher than LODO; per-dataset held-out-test-vs-LODO accuracy gaps span 1-25 points. To understand the gap, we analyze the LODO stability of a linear probe's per-feature classifier coefficients, defining a retention metric for sparse-autoencoder (SAE) features that flags dataset-dependent shortcuts. 28-44% of top SAE features are shortcuts across models, a dataset-identity classifier reaches 96.6%, and the dataset-identifying and safety-relevant subspaces partially overlap. Standard domain-generalization fixes such as adversarial training, subspace projection, sample reweighting, and class balancing do not close the gap. Finally, we show LODO-weighted SAE attributions filter dataset artifacts for more reliable per-prompt explanations. We release our framework at https://github.com/maxf-zn/prompt-mining so future prompt-attack classifiers can be evaluated under LODO alongside CV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。