arXiv:2511.04694cs.CLcs.AI2025-11ACL被引 6

让大模型学会按优先级处理多指令,提升决策可靠性。

Reasoning Up the Instruction Ladder for Controllable Language Models

  • 将指令优先级问题转化为推理任务,先思考再响应。
  • 在冲突场景下指令遵循准确率提升约20%。
  • 增强模型对越狱攻击的抵抗力,最高降低20%成功率。

随着大语言模型在现实决策中承担高风险角色,需在同一提示上下文中协调多个来源的相互竞争指令。建立指令层级机制,使高级别指令覆盖低优先级请求,对模型的可靠性与可控性至关重要。本文将指令层级解析重构为推理任务:模型需先思考用户提示与高层级指令间的关系,再生成回应。为此,我们构建了VerIH数据集,包含可验证答案的约束遵循任务,涵盖对齐与冲突的系统-用户指令。实验表明,轻量级强化学习结合VerIH能有效将模型的一般推理能力迁移至指令优先级判断。该方法在多个模型族上均取得一致提升,在指令遵循与层级基准测试中表现优异,冲突场景下准确率提升约20%。此外,模型在训练分布外的安全关键场景中也表现出更强鲁棒性,对越狱和提示注入攻击的防御能力显著增强,攻击成功率最高降低20%。结果表明,对指令层级的推理是提升AI可靠性的实用机制,通过针对性更新系统提示即可实现行为的可预测、可控与稳健变化。

原文摘要 · Abstract (English)

As large language model (LLM) based systems take on high-stakes roles in real-world decision-making, they must reconcile competing instructions from multiple sources within a single prompt context. Enforcing an instruction hierarchy, where higher-level directives override lower-priority requests, is critical to the reliability and control of LLMs. In this work, we reframe instruction hierarchy resolution as a reasoning task. The model must first "think" about the relationship between a given user prompt and higher-priority instructions before generating a response. To enable this capability, we construct VerIH, a training dataset of constraint-following tasks with verifiable answers, comprising aligned and conflicting system-user instructions. We show that lightweight reinforcement learning with VerIH effectively transfers general reasoning capabilities of models to instruction prioritization. Our method leads to consistent improvements across multiple model families on both instruction following and instruction hierarchy benchmarks, achieving ~20% absolute improvement in conflict setups. Our method also leads to improved alignment to safety-critical scenarios beyond the training distribution, exhibiting increased robustness against jailbreak and prompt injection, reducing absolute attack success rates by up to 20%. Our results establish reasoning over instruction hierarchies as a practical mechanism for improving AI reliability, where targeted updates to system prompts produce predictable, controllable, and robust changes in model behavior.

指令优先级模型可控性安全对抗推理机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。