让大模型按五级指令优先级执行,防恶意注入且减少错误拒绝。
Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization
- 设计五级指令层级,用引力加权偏好优化训练模型区分指令优先级。
- 在Llama-3.1-8B上实现10种配对优先级,错误拒绝率降至标准DPO一半。
- 适合需严格指令管控的高安全场景,如金融、医疗系统部署。
生产级大模型接收来自不同可信度来源的指令,但对所有词元一视同仁,存在结构漏洞,易受恶意提示注入攻击,且无法有效解决合法但冲突的指令。现有方法仅构建三至四级指令层级,将所有违规视为同等严重,且极少评估全部成对层级交互。本文形式化了k级指令层级问题,并实例化为k=5,形成十组成对优先关系,要求模型严格执行。提出引力加权直接偏好优化(GW-DPO),其样本偏置随冲突层级间结构距离变化,在线性或双边调度下,后者同时考虑特权差距与受害层级的特权。结合层级专用分隔符(Chen et al., 2025)和指令段嵌入(ISE; Wu et al., 2025),双边调度的GW-DPO在Llama-3.1-8B-Instruct上帕累托改进标准DPO与线性变体,提升宏观成对优先级遵循度,同时将过拒率控制在标准DPO的一半。消融实验表明ISE可校准拒绝阈值,五级训练与三级训练构成泛化与专精的权衡。
原文摘要 · Abstract (English)
Production LLMs receive instructions from sources with very different levels of trust, yet attend to every token with uniform architectural privilege. This is the structural vulnerability that enables malicious prompt injections and, more broadly, leaves models without a principled way to resolve conflicts between legitimate but competing instructions. A common training-based response is to teach models an explicit instruction hierarchy; existing approaches, however, formalize hierarchies of only three or four levels, treat all violations as equally severe, and rarely evaluate the full set of pairwise level interactions. We formalize a k-level instruction hierarchy problem and instantiate it for k=5, yielding ten pairwise priority relations that a compliant model must enforce. We then introduce Gravity-Weighted DPO (GW-DPO), a preference-optimization objective whose per-sample offset scales with the structural distance between conflicting levels under a linear or bilateral schedule, the latter weighting severity by both the privilege gap and the privilege of the victim level. Combined with hierarchy-specific delimiter tokens (Chen et al., 2025) and Instructional Segment Embeddings (ISE; Wu et al., 2025), GW-DPO with the bilateral schedule Pareto-improves over standard DPO and the linear variant on Llama-3.1-8B-Instruct, raising macro pairwise priority adherence while keeping over-refusal at half the standard DPO rate. Ablations isolate ISE as a refusal-threshold calibrator and recast five- versus three-level training as a generality-specialization tradeoff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。