arXiv:2506.09996cs.CLcs.CY2025-06NeurIPS被引 20

用流式监控提前拦截大模型有害输出,速度更快且准确度高。

From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring

  • 构建细粒度标注数据集FineHarm,支持逐标记训练。
  • 仅看输出前18%内容,宏F1超0.95,接近全量检测效果。
  • 适合需要低延迟安全防护的AI服务场景。

尽管安全对齐已广泛应用于大型语言模型(LLMs),但实际产品中服务商通常部署外部审核机制作为安全防线。现有审核器多采用全量检测,基于完整输出判断有害性,导致服务延迟高。近期工作关注部分检测,即在生成中途进行监测并早期终止有害输出,但直接将全量训练的审核器用于不完整输出,引入训练-推理不匹配,降低性能。本文提出数据与模型协同方案,原生支持部分检测:构建包含29,000个提示-响应对的FineHarm数据集,提供细粒度标注以支持逐标记训练;提出流式内容监控器(SCM),通过响应级和标记级双重监督训练,可随模型输出流实时判断有害性。实验表明,SCM平均仅需观察输出前18%的标记,即可获得超过0.95的宏F1分数,媲美全量检测;同时可作为伪有害性标注器提升安全对齐,使有害性得分优于DPO。

原文摘要 · Abstract (English)

Though safety alignment has been applied to most large language models (LLMs), LLM service providers generally deploy a subsequent moderation as the external safety guardrail in real-world products. Existing moderators mainly practice a conventional full detection, which determines the harmfulness based on the complete LLM output, causing high service latency. Recent works pay more attention to partial detection where moderators oversee the generation midway and early stop the output if harmfulness is detected, but they directly apply moderators trained with the full detection paradigm to incomplete outputs, introducing a training-inference gap that lowers the performance. In this paper, we explore how to form a data-and-model solution that natively supports partial detection. For the data, we construct FineHarm, a dataset consisting of 29K prompt-response pairs with fine-grained annotations to provide reasonable supervision for token-level training. Then, we propose the streaming content monitor, which is trained with dual supervision of response- and token-level labels and can follow the output stream of LLM to make a timely judgment of harmfulness. Experiments show that SCM gains 0.95+ in macro F1 score that is comparable to full detection, by only seeing the first 18% of tokens in responses on average. Moreover, the SCM can serve as a pseudo-harmfulness annotator for improving safety alignment and lead to a higher harmlessness score than DPO.

安全对齐流式检测大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。