构建可控基准测试隐藏PDF提示注入，验证混合检测方法有效性
CrackedPDFs: A Controlled Benchmark for Hidden Prompt Injection in PDFs
- 设计2.9万份生成PDF，含注入与对照样本，支持配对评估
- 混合检测器在测试集上达0.960 F1、0.998 ROC-AUC，准确率95.9%
- 揭示仅依赖文本的模型易被绕过，强调结构感知的重要性
文档型大模型系统常在防护机制前将PDF展平，可能丢失用户未见指令的证据。本文提出CrackedPDFs，一个针对隐藏提示注入的可控基准。包含从4,983个基础文档生成的29,322份PDF，其中9,774份含注入，19,548份为良性或匹配混淆样本。评估了PromptGuard、规则基线、仅结构学习模型及净化混合检测器。使用保留来源划分、成对良性-混淆对照、标签随机化检验和捷径审计。在2,919文档的独立测试集上，混合检测器达到0.960 F1、0.998 ROC-AUC和0.997 PR-AUC；在包含973个注入文件与973个匹配良性混淆样本的平衡子集上，分类准确率达95.9%，且在全部973对中将注入文件排在对应混淆文件之前。PromptGuard仅用提取文本时召回率低，仅结构学习模型在成对控制下表现弱，文本TF-IDF模型虽在独立测试中得满分，但无法通过捷径审计。结果表明，在受控成对评估下，文档感知的混合检测有效，但不保证广泛真实场景鲁棒性或跨家族泛化能力。
原文摘要 · Abstract (English)
Document-based LLM systems often flatten a PDF before guardrails inspect it. That step can discard evidence that an instruction was never visible to the user. We introduce CrackedPDFs, a controlled benchmark for hidden prompt injection in PDFs. The benchmark contains 29,322 generated PDFs from 4,983 base documents. It includes 9,774 injected files and 19,548 benign or matched-confounder files. We evaluate PromptGuard and a rule baseline. We also evaluate structural-only learned models and a sanitized hybrid detector. The evaluation uses held-out provenance splits and paired benign-confounder controls. It also uses label-shuffle checks and shortcut audits. On a 2,919-document held-out test set, the hybrid detector reaches 0.960 F1. ROC-AUC is 0.998 and PR-AUC is 0.997. On a balanced subset containing 973 injected PDFs and 973 matched benign confounders, the hybrid detector achieves 95.9% classification accuracy. Using score ordering, it ranks the injected member above its matched confounder in 100% of 973 pairs. PromptGuard has low recall when given extracted text only. Structural-only learned models are weak under paired controls. A text-only TF-IDF model reaches perfect held-out scores but fails shortcut audits. These results show that document-aware hybrid detection is useful under controlled paired evaluation. They do not show broad real-world robustness or reliable cross-family generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。