研究发现大模型安全性能不随代际提升，反而在第三代出现漏洞反弹。
Cross-Generational Transfer of Adversarial Attacks Reveals Non-Monotonic Safety Alignment in LLMs
- 用进化算法自动挖掘攻击，跨代测试模型安全性
- 第三代模型攻击成功率高达68.7%，高于前代和后继版本
- 动态探测揭示静态评测无法发现的安全退化现象
大型语言模型的安全对齐并非随代际单调提升。通过使用Google Gemma系列（7B-31B）四代模型，结合质量-多样性进化方法（MAP-Elites）作为自动化红队探测工具，我们发现Gemma 3（12B）的攻击成功率（ASR）达68.7% ± 5.7%（均值±标准差，3个种子），显著高于其前代Gemma 2（45.5% ± 7.2%；p = 0.030，配对自举检验）及后继Gemma 4（33.9% ± 1.8%）。跨代回放进化生成的攻击档案显示，其他代际的攻击在Gemma 3上成功率为44-46%，但在Gemma 4上仅14-18%，表明Gemma 4的安全提升可泛化至先前未遭遇的攻击分布。在8B裁判下，版权与网络犯罪漏洞在所有代际接近100%，但第二裁判审计（第6节）提示版权结果可能受裁判选择影响。虚假信息攻击成功率从Gemma 2的29%跃升至Gemma 3的99%，并在Gemma 4维持在77%，表明该退化未被完全修复。这些模式仅通过自适应、纵向探测显现，而静态基准无法捕捉。所有实验使用3个随机种子，统一自托管裁判；代码与数据集见https://github.com/bassrehab/red-queen。
原文摘要 · Abstract (English)
Safety alignment in LLMs does not improve monotonically across model generations. Studying four generations of Google's Gemma family (7B-31B) with quality-diversity evolution (MAP-Elites) as an automated red-teaming probe, we find that Gemma 3 (12B) exhibits 68.7% +/- 5.7% attack success rate (ASR; mean +/- std, 3 seeds), significantly higher than its predecessor Gemma 2 (45.5% +/- 7.2%; p = 0.030, paired bootstrap) and its successor Gemma 4 (33.9% +/- 1.8%). Replaying evolved attack archives across generations reveals that attacks from other generations transfer to Gemma 3 at 44-46% but only 14-18% to Gemma 4, indicating that Gemma 4's safety gains generalize beyond the attack distributions evolved against earlier generations. Under our 8B judge, copyright and cybercrime vulnerabilities register at near-100% across all generations, though a second-judge audit (Section 6) suggests the copyright result is sensitive to judge choice. Misinformation ASR jumps from 29% to 99% between Gemma 2 and Gemma 3 and remains elevated at 77% in Gemma 4, indicating the regression was not fully addressed. These patterns are invisible to static benchmarks and emerge only through adaptive, longitudinal probing. All experiments use 3 random seeds with a unified self-hosted judge; code and artifacts are available at https://github.com/bassrehab/red-queen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。