对比不同内容过滤位置与响应重写策略,发现响应阶段过滤+重写效果最佳。
Choosing Where and How to Moderate: End-to-End Trade-offs in Filter Placement and Response Rewriting

- 在响应阶段过滤并重写,可提升有用回复比例
- 输入+响应双阶段拦截能更低有害内容暴露率
- 适合关注实际部署中安全与延迟权衡的系统设计者
内容审核分类器通常孤立评估,但实际部署需决定干预位置及触发后的处理方式。本文采用两个端到端客户结果指标:有用性(展示且非有害的相关回复占比)和有害暴露率(展示有害回复占比),替代组件准确率。在人类标注产品基准和公开 ToxicChat 评测上,对比仅输入、仅响应、输入+响应硬拦截。在测试条件下,仅响应拦截在两组中均实现最高有用性;输入+响应拦截则降低有害暴露率。将仅响应拦截替换为响应重写,恢复了大部分被拦流量,且相同配置下有害暴露量与仅响应拦截相当(非等价结论)。探测路由相比大模型路由显著降低条件路由与生成时间,保持相近表现。聚焦输出审查显示,重写通过泛化触发语言保留良性意图并安全引导,但部分敏感领域输出仍遗漏关键安全信息。结果表明应基于具体部署场景的安全与延迟约束比较审核配置,而非通用规则。代码与公开数据见 https://github.com/microsoft/mod-frontier。
原文摘要 · Abstract (English)
Content-moderation classifiers are usually evaluated in isolation, but deployment requires choosing where to intervene and what follows a flag. We evaluate these choices using two end-to-end customer-outcome metrics rather than component accuracy: Usefulness, the fraction of turns with a shown, non-harmful, relevant response, and Harmful Exposure, the fraction with a shown harmful response. Latency and error rates are diagnostics. We compare Input only, Response only, and Input + response hard blocking on a human-labelled product benchmark and public ToxicChat evaluation. At the evaluated operating points, Response only achieves the highest filter-only Usefulness in both settings, while Input + response achieves lower Harmful Exposure. Replacing Response only blocking with Response + rewrite recovers most blocked traffic and yields the same observed Harmful Exposure count as Response only blocking for the selected configuration; this equality is not an equivalence result. Probe routing substantially reduces conditional route-and-generation time relative to LLM routing at comparable measured outcomes. A focused output review shows how rewrites balance filter passage with usefulness by generalizing triggering language while retaining benign intent and safe redirection; some sensitive-domain outputs nevertheless omit potentially safety-relevant support information. These results support comparing moderation configurations under deployment-specific safety and latency constraints rather than applying a universal placement rule. Code and public artifacts are available at https://github.com/microsoft/mod-frontier
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。