无需真实数据，用大模型生成测试用例，提升安全防护准确率。
A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection
- 用大模型自动生成多样化的越界提示，构建无数据依赖的测试集。
- 新方法在越界检测上优于传统规则，误报率更低。
- 适合开发阶段的AI安全防护，开源资源可直接复用。
大型语言模型易被用于越界任务。现有防护机制依赖人工标注或定制分类器，存在误报高、难适应、需真实数据等缺陷。本文提出一种无需真实数据的灵活防护开发方法：通过定性定义问题空间，让大模型生成多样化越界提示，构建合成数据集用于训练和评估防护系统。将任务建模为判断用户提示与系统指令的相关性，使防护机制能泛化至越狱和有害内容检测等场景。最后，我们开源了合成数据集和防护模型，为预生产环境下的模型安全研究提供支持。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are prone to off-topic misuse, where users may prompt these models to perform tasks beyond their intended scope. Current guardrails, which often rely on curated examples or custom classifiers, suffer from high false-positive rates, limited adaptability, and the impracticality of requiring real-world data that is not available in pre-production. In this paper, we introduce a flexible, data-free guardrail development methodology that addresses these challenges. By thoroughly defining the problem space qualitatively and passing this to an LLM to generate diverse prompts, we construct a synthetic dataset to benchmark and train off-topic guardrails that outperform heuristic approaches. Additionally, by framing the task as classifying whether the user prompt is relevant with respect to the system prompt, our guardrails effectively generalize to other misuse categories, including jailbreak and harmful prompts. Lastly, we further contribute to the field by open-sourcing both the synthetic dataset and the off-topic guardrail models, providing valuable resources for developing guardrails in pre-production environments and supporting future research and development in LLM safety.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。