测试对抗性提示发现大模型对齐机制在特定场景下有效但检测方式影响结果
Isolating LLM Alignment from Regex: Zero Coverage and Metric-Dependent Divergence Under Adversarial Mutation
- 禁用正则过滤后测试大模型对齐效果，使用对抗性提示探测
- 自然语言有害请求中对齐无额外防护作用，但对抗性提示下可识别拒绝
- 检测结果依赖评分方法，纯字符串匹配会漏检复杂拒绝行为
生产环境中的大模型应用常在模型前叠加正则过滤；此前研究发现，在启用Gemini后端时，增加实时对齐无显著覆盖提升。本文探究当语料库被设计为绕过正则表达式时该上限是否仍成立。引入$L_5$-no-regex（与$L_4$-real相同，仅禁用九模式过滤器），在三个子语料库（延续性、正则绕过、对齐隔离）上评估45个对抗性探针，经Gemini重述和PAIR增强至约1,555对探针运行，共5次重复。主子串分类器结果显示：$L_5$的阻断率为0%，在所有五类OWASP LLM Top-10类别中与$L_0$无差异（Δpp=0，p=1.00；Wilson上界<5%）。二级LLM裁判指标在PAIR变体上显示56%–100%阻断率（p<0.01），表明对齐能响应对抗性表述，但拒绝内容过于细微，无法通过子串匹配捕捉。子语料库差异预测不成立（p=1.00）。对齐贡献具度量依赖性：对自然语言有害请求，对齐未带来额外覆盖率；对对抗性表述，LLM裁判可识别子串分类器遗漏的拒绝。锁定语料、变异产物及导出脚本已公开供复现。
原文摘要 · Abstract (English)
Production LLM applications commonly stack a regex filter in front of model-side alignment; prior work found no measurable coverage gain from adding a live Gemini backend behind an active regex filter. We ask whether that ceiling holds when the corpus is \emph{designed to bypass the regex}. We introduce $L_5$-no-regex -- identical to $L_4$-real (Gemini-2.5-flash, token-budget cap, rate limit, output scrub) but with the nine-pattern filter disabled -- and evaluate it against $N{=}45$ adversarial probes across three sub-corpora (carry-forward, regex-bypass, alignment-isolate), amplified by Gemini paraphrase and PAIR to ${\sim}1{,}555$ probe-run pairs over $N{=}5$ replications. Under the primary substring classifier, H1 is refuted: $L_5$ block rate is $0,%$ across all five OWASP LLM Top-10 categories ($Δ\text{pp}{=}0$ vs.\ $L_0$, $p{=}1.00$; Wilson upper bound ${<}5,%$). A secondary LLM-judge metric on PAIR variants shows $56$--$100,%$ block rates ($p{<}0.01$), revealing alignment does respond to adversarially-framed probes -- but produces refusals too nuanced for substring matching. The sub-corpus differential prediction is not supported ($p{=}1.00$). Alignment's contribution is \emph{metric-dependent}: on natural-language harmful-request probes, it adds zero observed coverage beyond the regex; on adversarially-framed variants, an LLM judge detects refusals the substring classifier misses. The locked corpus, mutation artifacts, and export scripts are released for replication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。