KidsNanny用双阶段多模态分析,高效识别含文字的儿童安全威胁内容。
KidsNanny: A Two-Stage Multimodal Content Moderation Pipeline Integrating Visual Classification, Object Detection, OCR, and Contextual Reasoning for Child Safety
- 先用视觉模型快速筛查,再结合OCR与大模型进行上下文推理
- 整体120毫秒内达到86.16%的F1值,优于多个现有模型
- 对嵌入文字的安全威胁可实现100%召回率,适合敏感内容检测场景
我们提出KidsNanny,一种用于儿童安全的两阶段多模态内容审核架构。第一阶段结合视觉变压器(ViT)与目标检测器进行视觉筛查,耗时11.7毫秒;输出以文本形式传递至第二阶段,该阶段执行OCR并调用基于70亿参数的语言模型进行上下文推理,总耗时120毫秒。在UnsafeBench性相关类别(1,054张图像)上评估,仅视觉阶段准确率为80.27%,F1为85.39%;完整管道达到81.40%准确率和86.16% F1,显著优于ShieldGemma-2(64.80%准确率,1,136毫秒)和LlavaGuard(80.36%准确率,4,138毫秒)。针对文本感知能力测试,在257张图文混合样本和44张纯文本样本上,纯文本子集表现优异:100%召回率(25/25阳性),75.76%精确率。结果表明,专用OCR推理在低延迟下可能更擅长捕捉文本嵌入威胁,但小样本限制了泛化性。本研究旨在推动高效多模态内容审核在儿童安全领域的应用。
原文摘要 · Abstract (English)
We present KidsNanny, a two-stage multimodal content moderation architecture for child safety. Stage 1 combines a vision transformer (ViT) with an object detector for visual screening (11.7 ms); outputs are routed as text not raw pixels to Stage 2, which applies OCR and a text based 7B language model for contextual reasoning (120 ms total pipeline). We evaluate on the UnsafeBench Sexual category (1,054 images) under two regimes: vision-only, isolating Stage 1, and multimodal, evaluating the full Stage 1+2 pipeline. Stage 1 achieves 80.27% accuracy and 85.39% F1 at 11.7 ms; vision-only baselines range from 59.01% to 77.04% accuracy. The full pipeline achieves 81.40% accuracy and 86.16% F1 at 120 ms, compared to ShieldGemma-2 (64.80% accuracy, 1,136 ms) and LlavaGuard (80.36% accuracy, 4,138 ms). To evaluate text-awareness, we filter two subsets: a text+visual subset (257 images) and a text-only subset (44 images where safety depends primarily on embedded text). On text-only images, KidsNanny achieves 100% recall (25/25 positives; small sample) and 75.76% precision; ShieldGemma-2 achieves 84% recall and 60% precision at 1,136 ms. Results suggest that dedicated OCR-based reasoning may offer recall-precision advantages on text-embedded threats at lower latency, though the small text-only subset limits generalizability. By documenting this architecture and evaluation methodology, we aim to contribute to the broader research effort on efficient multimodal content moderation for child safety.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。