用少量样本生成高质量训练数据,让小模型高效守护自定义规则。
BARRED: Synthetic Training of Custom Policy Guardrails via Asymmetric Debate

- 通过分解任务维度并多智能体辩论,自动生成高保真合成数据。
- 小模型在合成数据上微调后性能超越主流大模型和专用安全模型。
- 无需人工标注,适合需要定制化安全策略的场景。
部署自定义策略的防护机制仍具挑战:通用安全模型难以捕捉任务特定需求,而提示大语言模型存在边界案例表现不一且推理成本高。训练专属分类器虽兼顾准确率与效率,但需大量标注数据,获取成本高昂。本文提出BARRED(边界对齐精炼通过反思与辩论),仅需任务描述和少量未标注样例,即可生成忠实且多样化的合成训练数据。该方法将领域空间分解为多个维度以确保覆盖全面,并利用多智能体辩论验证标签正确性,形成高保真训练语料。在多种自定义策略上的实验表明,基于合成数据微调的小语言模型持续优于当前最先进的专有大模型(包括推理型模型)及专用防护模型。消融实验确认,维度分解与辩论验证均对保障多样性与标签真实性至关重要。BARRED框架摆脱了对大规模人工标注的依赖,为精准定制化防护提供了可扩展解决方案。
原文摘要 · Abstract (English)
Deploying guardrails for custom policies remains challenging, as generic safety models fail to capture task-specific requirements, while prompting LLMs suffers from inconsistent boundary-case performance and high inference costs. Training custom classifiers achieves both accuracy and efficiency, yet demands substantial labeled data that is costly to obtain. We present BARRED (Boundary Alignment Refinement through REflection and Debate), a framework for generating faithful and diverse synthetic training data using only a task description and a small set of unlabeled examples. Our approach decomposes the domain space into dimensions to ensure comprehensive coverage, and employs multi-agent debate to verify label correctness, yielding a high-fidelity training corpus. Experiments across diverse custom policies demonstrate that small language models finetuned on our synthetic data consistently outperform state-of-the-art proprietary LLMs (including reasoning models) and dedicated guardrail models. Ablation studies confirm that both dimension decomposition and debate-based verification are critical for ensuring the diversity and label fidelity required for effective fine-tuning. The BARRED framework eliminates the reliance on extensive human annotation, offering a scalable solution for accurate custom guardrails.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。