用规则对齐与分层标签提升多模态内容审核的准确性和可解释性。
Towards Trustworthy Multimodal Moderation via Policy-Aligned Reasoning and Hierarchical Labeling
- 分层处理:先用轻量模型筛除安全内容,再用强模型进行细粒度风险分类。
- 引入规则提示和软边距奖励,使模型更符合政策变化并减少相近类误判。
- 适合需要高可信度、可解释性的平台内容安全团队使用。
社交平台重塑了信息传播方式,但也加速了有害及违规内容的扩散。为确保大规模下的安全与合规,审核系统需超越效率,实现高精度与可解释性。然而,现有方法多依赖噪声标签驱动学习,缺乏与审核规则的对齐,决策过程不透明,阻碍人工复核。为此,我们提出分层守护(Hi-Guard)多模态审核框架,引入规则对齐决策范式。'分层'体现在两方面:(1) 分层审核流程,由轻量级二分类模型先过滤安全内容,再由强模型完成细粒度风险分类;(2) 第二阶段采用路径式分类的层级标签体系,从粗到细逐级划分风险等级。为确保与动态政策对齐,Hi-Guard将规则定义直接嵌入模型提示。为进一步提升结构化预测与推理能力,提出多级软边距奖励,并通过组相对策略优化(GRPO)进行训练,惩罚语义相近类别的误判,提升解释质量。大量实验与真实部署验证表明,Hi-Guard在分类准确率、泛化能力和可解释性上均表现卓越,为构建可扩展、透明、可信的内容安全系统铺平道路。代码已开源:https://github.com/lianqi1008/Hi-Guard。
原文摘要 · Abstract (English)
Social platforms have revolutionized information sharing, but also accelerated the dissemination of harmful and policy-violating content. To ensure safety and compliance at scale, moderation systems must go beyond efficiency and offer accuracy and interpretability. However, current approaches largely rely on noisy, label-driven learning, lacking alignment with moderation rules and producing opaque decisions that hinder human review. Therefore, we propose Hierarchical Guard (Hi-Guard), a multimodal moderation framework that introduces a new policy-aligned decision paradigm. The term "Hierarchical" reflects two key aspects of our system design: (1) a hierarchical moderation pipeline, where a lightweight binary model first filters safe content and a stronger model handles fine-grained risk classification; and (2) a hierarchical taxonomy in the second stage, where the model performs path-based classification over a hierarchical taxonomy ranging from coarse to fine-grained levels. To ensure alignment with evolving moderation policies, Hi-Guard directly incorporates rule definitions into the model prompt. To further enhance structured prediction and reasoning, we introduce a multi-level soft-margin reward and optimize with Group Relative Policy Optimization (GRPO), penalizing semantically adjacent misclassifications and improving explanation quality. Extensive experiments and real-world deployment demonstrate that Hi-Guard achieves superior classification accuracy, generalization, and interpretability, paving the way toward scalable, transparent, and trustworthy content safety systems. Code is available at: https://github.com/lianqi1008/Hi-Guard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。