研究大模型代码生成中行业提示对安全漏洞的影响，发现行业术语不提升安全性。
SecDrift: Measuring Sector-Conditioned Security Drift in AI-Generated Code
- 用领域术语提示生成代码，对比静态分析漏洞率变化。
- 行业提示下漏洞率仅14.0%，与中性提示差异不显著（p=0.24）。
- 模型选择比提示设计对安全影响更大，适合关注生成安全的开发者。
大型语言模型在关键基础设施代码生成中应用日益广泛，但领域特定提示的安全影响尚不明确。本文提出SecDrift基准，测量行业条件下的安全漂移：当提示基于行业背景而非中性基准时，静态分析漏洞率的变化。评估了7个LLM（6个生成可分析代码），覆盖8个CISA关键基础设施领域和9类常见漏洞（CWE），共5355次评估。采用五维转换，在任务固定前提下仅替换领域术语。行业提示看似更安全（14.0% vs. 11.4%，-2.7pp），但该差异不显著（Fisher精确检验 p=0.24，Cohen's h=-0.08），且是两个CWE类别（CWE-502、CWE-22）的合成结果；剔除后差距消失甚至逆转（+0.4pp，p=1.00）。混合效应逻辑回归表明，领域身份非调节因素，唯一可检测的条件效应仅存在于这两类漏洞。8个领域均未出现显著漂移（|h|<0.15）。对非CISA领域（电商、在线教育）的对照实验重现了相似漏洞率（10.5% vs. 11.4%，p=0.63），说明小规模模式源于通用行业表述，非关键基础设施特异性。相比之下，模型选择影响显著且一致：全输出模型漏洞率在11.6%至16.1%间波动，且跨条件稳定。模型选型比提示设计更具可靠性。研究发布框架、提示、生成代码、结论、人工验证结果及分析脚本。
原文摘要 · Abstract (English)
LLMs are increasingly used for code generation in critical infrastructure, yet the security effect of domain-specific prompting is understudied. We present SecDrift, a benchmark measuring sector-conditioned security drift: the change in static-analysis vulnerability rates when prompts are conditioned on industry contexts versus neutral baselines. We evaluate 7 LLMs (6 producing analyzable code) across 8 CISA critical infrastructure sectors and 9 CWE categories with 5 replicates (5,355 evaluations), using a 5-dimension transformation with a matched-baseline condition that holds the task fixed while substituting only domain terminology. Industry prompts naively appear more secure (14.0% vs. 11.4%, -2.7pp), but the gap is not statistically significant (Fisher's exact p = 0.24, Cohen's h = -0.08) and is a composition artifact of two CWE categories: excluding CWE-502 and CWE-22 eliminates and slightly reverses it (+0.4pp, p = 1.00). A mixed-effects logistic regression confirms sector identity is not a moderator and localizes the only detectable condition effect to those two vulnerability types. 0 of 8 sectors show drift distinguishable from baseline, corrected or uncorrected (|h| < 0.15). A placebo on two non-CISA sectors (e-commerce, online education) reproduces the CISA industry rate almost exactly (10.5% vs. 11.4%, p = 0.63): the small pooled pattern reflects generic industry-framing specificity, not critical-infrastructure identity. In contrast, model selection has a large and consistent effect: among full-output models vulnerability rates range from 11.6% to 16.1%, and these differences persist across conditions. Model choice, not prompt framing, is the more reliable security lever. We release the framework, prompts, generated code, findings, human-validation verdicts, and analysis scripts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。