提出可区分每轮贡献的攻击评分框架,让大模型更精准地学会有效越狱。
Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking

- 用留一法掩码评估每轮对越狱成功的真实贡献
- 失败轨迹中根据危害性与相关性惩罚关键中间轮
- 兼顾攻击效率与安全防御对齐,提升越狱成功率25%
在多轮对话中部署大语言模型易受越狱攻击,此类攻击将有害意图分散至看似无害的各轮对话中。现有基于训练的多轮越狱方法依赖全局结果信号对所有轮次统一奖励,但发现各轮贡献存在非均匀性、阶段依赖性和目标特异性。这种粗粒度监督引发信用分配问题,导致成功轨迹中冗余轮次被过度奖励,失败轨迹中有用中间轮被低估。为此,我们提出TRACE——一种面向强化学习的多轮越狱任务的轮次感知信用分配框架。对于成功轨迹,通过留一法语义掩码估计每轮贡献;对于失败轨迹,依据提示危害性与语义相关性分配惩罚,并引入局部拒绝感知惩罚。此外,我们将攻击侧信用信号复用于多轮防御对齐。在开源与闭源目标上的大量实验表明,TRACE在有效性、可迁移性与效率方面表现优异,相比最强基线相对提升约25%的攻击成功率,同时在用于防御对齐时改善了安全性与可用性的平衡。
原文摘要 · Abstract (English)
Deploying LLMs in multi-turn dialogues facilitates jailbreak attacks that distribute harmful intent across seemingly benign turns. Recent training-based multi-turn jailbreak methods learn long-horizon attack strategies from interaction feedback, but often rely on coarse trajectory-level outcome signals that broadcast uniformly to every turn. However, we find that turn-level contributions in multi-turn jailbreaking are non-uniform, phase-dependent, and target-specific. Such coarse outcome supervision induces a credit assignment problem, leading to over-rewarding redundant turns in successful trajectories and under-crediting useful intermediate turns in failed ones. To address this, we propose TRACE, a turn-aware credit assignment framework for reinforcement learning (RL)-based multi-turn jailbreaking. For successful trajectories, TRACE estimates turn-level contributions via leave-one-turn-out semantic masking; for failed ones, TRACE assigns penalties based on prompt harmfulness and semantic relevance, with an additional local refusal-aware penalty. Furthermore, we reuse the attack-side credit signal for multi-turn defense alignment. Extensive experiments on open-source and closed-source targets show that TRACE achieves strong overall performance in effectiveness, transferability, and efficiency, yielding about a 25% relative improvement in attack success rate over the strongest RL baseline while also improving the safety-utility balance when reused for defense alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。