让大模型按优先级正确处理多指令冲突,提升实际应用中的行为一致性。
Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistency

- 将指令冲突建模为约束满足问题,用求解器引导推理
- 在真实场景下显著提升多指令下的任务执行准确率
- 适合需要可靠多任务协作的复杂应用系统
大型语言模型在现实应用中常需同时响应来自不同来源、权威性不同的多种指令,包括系统策略、用户请求、工具输出和检索内容。现有指令层级研究多聚焦对抗攻击,忽略了真实场景中常见的良性指令冲突。在此类场景下,模型不仅要避免安全违规,还需保持任务有效性与行为一致性。本文提出神经符号层次对齐(NSHA),通过显式建模与强制执行指令优先级来实现层次化指令遵循。推理时,引入求解器引导推理,将指令解析转化为约束满足问题,使模型能在层级约束下推导出最一致的可执行指令集;训练时,利用自动生成的监督信号将求解器决策蒸馏进模型参数。我们在规则遵循、任务执行、工具使用和安全性等多个任务上评估,涵盖单轮与多轮交互,结果表明,NSHA在指令冲突下性能显著提升,且在基准设置中保持竞争力。
原文摘要 · Abstract (English)
Large language models increasingly operate under multiple instructions from heterogeneous sources with different authority levels, including system policies, user requests, tool outputs, and retrieved context. While prior work on instruction hierarchy highlights the importance of respecting instruction priorities, it mainly focuses on adversarial attacks and overlooks the benign but common instruction conflicts that arise in real-world applications. In such settings, models must not only avoid security violations but also preserve task utility and behavioral consistency when instructions partially or implicitly conflict. We propose Neuro-Symbolic Hierarchical Alignment (NSHA) for hierarchical instruction-following by explicitly modeling and enforcing instruction priorities. At inference time, we introduce solver-guided reasoning that formulates instruction resolution as a constraint satisfaction problem, enabling the model to derive a maximally consistent set of applicable instructions under hierarchical constraints. At training time, NSHA distills solver-based decisions into model parameters using automatically constructed supervision. We evaluate our approach on rule following, task execution, tool use, and safety, covering both single-turn and multi-turn interactions, and show that NSHA significantly improves performance under such conflicts while maintaining competitive utility in reference settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。