用过程奖励优化多轮攻击，提升大模型越狱成功率。
TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewards
- 将多轮越狱建模为强化学习问题，以最终输出危害性为奖励。
- 引入两个过程奖励：避免触发拒绝机制、引导回复向有害内容靠拢。
- 在多个模型和数据集上显著提升攻击成功率，适合安全测试人员。
大语言模型广泛应用，但仍易受多轮越狱攻击威胁其安全部署。现有方法多依赖逐轮优化，难以学习长期攻击策略。为此，本文将该任务建模为多轮强化学习问题，直接以最终回复的危害性作为结果奖励。为缓解结果奖励稀疏问题，提出TROJail，引入两个过程奖励：（1）惩罚过度有害的提示，防止触发模型拒绝机制；（2）鼓励中间回复语义向目标有害内容靠拢，并将其融入优势估计。实验表明，该方法在多个模型与基准测试中均提升了攻击成功率，验证了有效性。代码已公开于https://github.com/xxiqiao/TROJail。警告：本文包含有害内容示例。
原文摘要 · Abstract (English)
Large language models have seen widespread adoption, yet they remain vulnerable to multi-turn jailbreak attacks, threatening their safe deployment. This has led to the task of training automated multi-turn attackers to probe model safety vulnerabilities. However, existing approaches typically rely on turn-level optimization, which is insufficient for learning long-term attack strategies. To bridge this gap, we formulate this task as a multi-turn reinforcement learning problem, directly optimizing the harmfulness of the final-turn response as the outcome reward. To address the sparse supervision of the outcome reward, we introduce TROJail, which employs two process rewards to evaluate the utility of intermediate prompts and integrate them into advantage estimation. These rewards (1) penalize overly harmful prompts that trigger the model's refusal mechanism, and (2) encourage steering the semantic relevance of responses toward the targeted harmful content. Experimental results show improved attack success rates across multiple models and benchmarks, highlighting the effectiveness of our approach. The code is available at https://github.com/xxiqiao/TROJail. Warning: This paper contains examples of harmful content.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。