构建对抗性数据集，提升大模型在指令冲突下的优先级判断能力。
IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs
- 设计对抗性训练数据集，通过强化学习优化指令优先级策略。
- 在16项评测中指令层级鲁棒性提升10.0%，安全行为降低至0.7%。
- 适合关注模型安全、对抗攻击防御的研究者与开发者。
指令层级（IH）定义了大模型在系统、开发者、用户和工具指令冲突时的优先顺序，是应对越狱攻击、系统提示提取和代理式提示注入的关键机制。然而，训练鲁棒的指令层级行为极具挑战：其失败常与指令遵循失败混淆，冲突情境复杂，且模型可能习得过度拒绝等捷径行为。本文提出IH-Challenge，一个用于强化学习的训练数据集。将GPT-5-Mini在该数据集上进行微调，并结合在线对抗样本生成，使指令层级鲁棒性在16个分布内、分布外及人工红队测试中平均提升10.0%（从84.1%升至94.1%），不安全行为从6.6%降至0.7%，同时保持通用安全性评估中的帮助性，且在内部静态代理提示注入测试中达到饱和表现，能力退化极小。数据集已开源，以支持后续研究。
原文摘要 · Abstract (English)
Instruction hierarchy (IH) defines how LLMs prioritize system, developer, user, and tool instructions under conflict, providing a concrete, trust-ordered policy for resolving instruction conflicts. IH is key to defending against jailbreaks, system prompt extractions, and agentic prompt injections. However, robust IH behavior is difficult to train: IH failures can be confounded with instruction-following failures, conflicts can be nuanced, and models can learn shortcuts such as overrefusing. We introduce IH-Challenge, a reinforcement learning training dataset, to address these difficulties. Fine-tuning GPT-5-Mini on IH-Challenge with online adversarial example generation improves IH robustness by +10.0% on average across 16 in-distribution, out-of-distribution, and human red-teaming benchmarks (84.1% to 94.1%), reduces unsafe behavior from 6.6% to 0.7% while improving helpfulness on general safety evaluations, and saturates an internal static agentic prompt injection evaluation, with minimal capability regression. We release the IH-Challenge dataset (https://huggingface.co/datasets/openai/ih-challenge) to support future research on robust instruction hierarchy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。