提出稳定生成流网络，提升大模型红队攻击的多样性与鲁棒性。
Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balance
- 用成对比较替代归一化常数估计，降低训练不稳定性。
- 在多个数据集上攻击成功率超基线30%以上，且生成样本更多样。
- 适合研究模型安全、对抗攻击的学者与工程师使用。
大语言模型红队测试是主动发现模型漏洞、保障安全的关键环节。有效且多样的攻击方法至关重要，但二者兼得极具挑战。生成流网络（GFN）虽具潜力，却因训练不稳定和模式崩溃而受限，尤其在红队任务中，不稳定的奖励信号会加剧模式崩溃。本文提出稳定生成流网络（S-GFN），通过成对比较避免对归一化常数 $Z$ 的估计，显著降低训练波动；引入鲁棒掩码机制应对噪声奖励，并设计流畅性稳定器防止模型陷入生成无意义文本的局部最优。S-GFN在保持GFN最优策略的同时实现更稳定训练，在多种设置下均展现出压倒性的攻击性能与更高的多样性。代码已开源：https://github.com/kmc0207/Stable-GFN。
原文摘要 · Abstract (English)
Large Language Model (LLM) Red-Teaming, which proactively identifies vulnerabilities of LLMs, is an essential process for ensuring safety. Finding effective and diverse attacks in red-teaming is important, but achieving both is challenging. Generative Flow Networks (GFNs) that perform distribution matching are promising methods, but they are notorious for training instability and mode collapse. In particular, unstable rewards in red-teaming accelerate mode collapse. We propose Stable-GFN (S-GFN), which eliminates partition function $Z$ estimation in GFN and reduces training instability. S-GFN avoids $Z$ estimation through pairwise comparisons and employs a robust masking methodology against noisy rewards. Additionally, we propose a fluency stabilizer to prevent the model from getting stuck in local optima that produce gibberish. S-GFN provides more stable training while maintaining the optimal policy of GFN. We demonstrate the overwhelming attack performance and diversity of S-GFN across various settings. Our code can be found in https://github.com/kmc0207/Stable-GFN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。