探针检测恶意输入在新数据上失效，因只学表面模式而非真正危害性。
False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize
- 用简单n-gram方法即可达到与探针相当的检测效果
- 探针实际学习的是指令格式和触发词等表面特征
- 适合关注大模型安全评估局限性的研究者
大型语言模型（LLMs）可能响应有害指令，引发严重安全问题。近期研究采用探针方法分析模型内部表示中恶意与正常输入的可分性，并提出用此类方法进行安全检测。本文系统重新审视该范式，基于其分布外性能差，推测探针学习的是表层模式而非语义危害性。通过控制实验验证该假设，发现探针主要学习指令模式和触发词。研究循序渐进：先展示简单n-gram方法表现相当，再使用语义清洗数据集进行控制实验，最后分析模式依赖关系。结果揭示当前探针方法存在虚假安全感，强调需重构模型与评估协议，本文亦提供进一步讨论以引导负责任的研究方向。项目已开源：https://github.com/WangCheng0116/Why-Probe-Fails。
原文摘要 · Abstract (English)
Large Language Models (LLMs) can comply with harmful instructions, raising serious safety concerns despite their impressive capabilities. Recent work has leveraged probing-based approaches to study the separability of malicious and benign inputs in LLMs' internal representations, and researchers have proposed using such probing methods for safety detection. We systematically re-examine this paradigm. Motivated by poor out-of-distribution performance, we hypothesize that probes learn superficial patterns rather than semantic harmfulness. Through controlled experiments, we confirm this hypothesis and identify the specific patterns learned: instructional patterns and trigger words. Our investigation follows a systematic approach, progressing from demonstrating comparable performance of simple n-gram methods, to controlled experiments with semantically cleaned datasets, to detailed analysis of pattern dependencies. These results reveal a false sense of security around current probing-based approaches and highlight the need to redesign both models and evaluation protocols, for which we provide further discussions in the hope of suggesting responsible further research in this direction. We have open-sourced the project at https://github.com/WangCheng0116/Why-Probe-Fails.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。