arXiv:2409.13733cs.CLcs.AI2024-09被引 8

让大模型学会按开发者设定的角色和规则响应,提升安全可控性。

RNR: Teaching Large Language Models to Follow Roles and Rules

  • 通过自动化生成角色与规则数据,训练模型理解复杂系统指令
  • 在Alpaca和Ultrachat上规则遵守率提升超25%,且不影响原有指令遵循能力
  • 适合需要高安全性、可定制行为的大模型部署场景

指令微调(IFT)通过监督学习赋予大语言模型遵循用户指令的能力,但现有开源数据集训练的模型通常仅能响应用户指令,难以遵循开发者设定的复杂角色与规则(即系统提示)。这一能力对实际部署至关重要,可确保模型在预设规范内安全交互。为此,我们提出 model,一个自动化数据生成流程,从现有IFT指令中衍生出多样化的角色与规则及对应响应。该数据可用于训练更懂系统提示的模型。我们在新构建的角色与规则遵循基准上评估模型,同时测试标准指令遵循任务和通用NLP任务。实验表明,该框架显著提升模型的角色与规则遵循能力,在Alpaca和Ultrachat数据集上规则完全遵守率(pass-rate)提升超过25%。此外,模型在主流指令遵循基准上无性能下降。

原文摘要 · Abstract (English)

Instruction fine-tuning (IFT) elicits instruction following capabilities and steers the behavior of large language models (LLMs) via supervised learning. However, existing models trained on open-source IFT datasets only have the ability to follow instructions from users, and often fail to follow complex role and rules specified by developers, a.k.a. system prompts. The ability to follow these roles and rules is essential for deployment, as it ensures that the model safely interacts with users within developer defined guidelines. To improve such role and rule following ability, we propose \model, an automated data generation pipeline that generates diverse roles and rules from existing IFT instructions, along with corresponding responses. This data can then be used to train models that follow complex system prompts. The models are evaluated on our newly created benchmarks for role and rule following ability, as well as standard instruction-following benchmarks and general NLP tasks. Our framework significantly improves role and rule following capability in LLMs, as evidenced by over 25% increase in pass-rate on rule adherence, i.e. following all requirements, in our experiments with the Alpaca and Ultrachat datasets. Moreover, our models achieves this increase without any regression on popular instruction following benchmarks.

大模型角色遵循系统提示指令微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。