检测大模型在输出前泄露凭证的行为,提前预警攻击。
Caught in the Act(ivation): Toward Pre-Output and Multi-Turn Detection of Credential Exfiltration by LLM Agents

- 用激活特征提前检测敏感信息访问,不依赖输出文本。
- 通过蜜罐与分段置信预测提升检测精度,识别隐蔽攻击。
- 追踪多轮对话中的泄漏累积量,发现单轮漏检的攻击。
大模型代理常将敏感凭证与不可信检索内容置于同一上下文窗口,为间接提示注入攻击创造了直接路径,导致凭证泄露。本文从三个互补方向研究这一漏洞:第一,探究激活探测是否能在输出令牌生成前检测到凭证访问;第二,构建基于格式特性的字符蜜罐,并利用分段置信预测进行校准检测;第三,将多轮泄露视为累积信息流问题,通过对话轮次间的泄漏预算跟踪来识别攻击。在开源权重模型上的受控实验表明,激活特征能以高准确率区分正常与凭证请求提示,包括在未见过的编码变换下依然有效。在小型合成多轮测试中,累积计数方法成功捕获了单轮检测遗漏的攻击。尽管结果仍属初步——多轮基准为内部小规模数据集,激活方法需白盒访问,且泄漏估计仅为实用信号而非形式上界——但结果表明,防御应结合输出前监控、校准蜜罐检测与时间维度泄漏追踪,而非仅依赖文本级输出过滤。
原文摘要 · Abstract (English)
LLM agents often place sensitive credentials in the same context window as untrusted retrieved content, creating a direct path for indirect prompt injection to induce credential exfiltration. We study this failure mode through three complementary defenses. First, we ask whether activation probes can detect credential access before output tokens are emitted. Second, we construct honeytokens from format-specific character models and calibrate detection with split conformal prediction. Third, we treat multi-turn exfiltration as a cumulative information-flow problem and track an estimated leakage budget across conversation turns. In controlled experiments on open-weight models, activation features separate benign and credential-seeking prompts with high accuracy, including under held-out encoding transformations. In a small synthetic multi-turn suite, cumulative accounting detects attacks that per-turn detectors miss. These results are preliminary: the multi-turn benchmark is in-house and small, the activation method requires white-box access, and the information estimator provides a practical signal rather than a formal upper bound. Still, the results suggest that credential-exfiltration defenses should combine pre-output monitoring, calibrated canary detection, and temporal leakage accounting rather than relying only on text-level output filters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。