开源大模型在不同领域安全响应差异巨大，且难以预测。
Unpredictable Safety: Domain-Dependent Compliance and the Transparency Gap in Open-Weight LLMs
- 通过双语境测试（分析/操作）发现模型对同一行为响应不一
- 安全合规率从14.7%到85.7%，跨领域差距达71个百分点
- 模型对有害请求的响应受表述方式影响，部署者难以察觉
我们系统研究了开源大模型在7个伦理领域中的领域依赖性安全行为：在4,200次交互中对5个模型（12B–70B）进行7项标准化实验，采用双判官验证。每种情境均以分析性框架（识别危害）和操作性框架（协助实施危害）测试，发现合规率从人类贩运的14.7%到监控设计的85.7%不等，跨度达71个百分点，置信区间无重叠。同一模型（Mistral Nemo 12B）在监控设计请求中全响应（100%），但对贩运仅26.7%响应。这种不可预测性源于技术表述绕过机制——将有害请求重构为工程问题可绕过安全训练，且无外部信号提示拒绝阈值变化。领域内异质性高达84.4个百分点，表明即使在同领域也无法可靠预测安全行为。在五款前沿闭源模型（GPT-4.1/5.2，Claude Haiku/Sonnet/Opus 4.x；n=4,163）上通过GitHub Copilot CLI复现该结果，领域分层模式一致，仅绝对水平降低，低编码领域（科学造假、监控）仍最宽容。结果表明当前安全机制缺乏可信部署所需的透明性与一致性。
原文摘要 · Abstract (English)
We present a systematic study of domain-dependent safety behavior in open-weight LLMs: 7 standardized experiments across 7 ethical domains, testing 5 models (12B--70B) in 4,200 interactions with dual-judge validation. Using a dual-condition methodology, each scenario tested in both an analytical framing (identify the harm) and an operational framing (help commit the harm), we find compliance rates vary from 14.7% (human trafficking) to 85.7% (surveillance design), a 71-percentage-point span with non-overlapping cluster-bootstrapped 95% CIs. Trustworthy deployment requires predictable safety behavior, yet we find compliance is highly context-dependent: the same model (Mistral Nemo 12B) provides surveillance designs in 100% of requests but assists with trafficking in only 26.7%. This unpredictability is opaque to deployers: the technical framing bypass, where harmful requests reframed as engineering problems override safety training without any external signal that refusal thresholds have shifted. Within-domain heterogeneity reaches 84.4pp, meaning safety behavior cannot be predicted even at the domain level. A replication on five frontier closed models (GPT-4.1/5.2, Claude Haiku/Sonnet/Opus 4.x; n=4,163 responses) accessed via the GitHub Copilot CLI deployed-product surface reproduces the same domain stratification, attenuated in absolute level but identical in shape, with the two low-codification domains (science fraud, surveillance) again the most permissive. These results show that current safety mechanisms lack the transparency and consistency required for trustworthy AI deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。