轻量模型主处理，关键时刻才调用大模型，提升直播内容检测效率。
StreamSense: Streaming Social Task Detection with Selective Vision-Language Model Routing
- 轻量编码器处理大部分数据，复杂情况才触发大模型
- 仅在必要时调用视觉语言模型，平均延迟和算力更低
- 支持实时判断不足时暂不决策，适合直播场景
直播平台需要实时感知社交信号，利用视频、文本、音频的局部且异步信息。我们提出 StreamSense，一种流式检测系统，将轻量级流式编码器与可选的视觉-语言模型（VLM）专家路由结合。StreamSense 以轻量编码器处理多数时间戳，对困难或模糊案例调用 VLM，对上下文不足的情况延迟决策。编码器通过跨模态对比损失对齐视听线索与文本信号，并使用 IoU 加权损失降低重叠不佳目标段的权重，缓解段边界处标签干扰。我们在多个社交流媒体检测任务（如情感分类、仇恨内容审核）上评估，结果表明，相比纯 VLM 流式处理，StreamSense 在保持更高准确率的同时，仅偶尔调用 VLM，显著降低平均延迟与计算开销。结果表明，选择性升级与延迟决策是理解流式社交任务的有效机制。代码已开源于 GitHub。
原文摘要 · Abstract (English)
Live streaming platforms require real-time monitoring and reaction to social signals, utilizing partial and asynchronous evidence from video, text, and audio. We propose StreamSense, a streaming detector that couples a lightweight streaming encoder with selective routing to a Vision-Language Model (VLM) expert. StreamSense handles most timestamps with the lightweight streaming encoder, escalates hard/ambiguous cases to the VLM, and defers decisions when context is insufficient. The encoder is trained using (i) a cross-modal contrastive term to align visual/audio cues with textual signals, and (ii) an IoU-weighted loss that down-weights poorly overlapping target segments, mitigating label interference across segment boundaries. We evaluate StreamSense on multiple social streaming detection tasks (e.g., sentiment classification and hate content moderation), and the results show that StreamSense achieves higher accuracy than VLM-only streaming while only occasionally invoking the VLM, thereby reducing average latency and compute. Our results indicate that selective escalation and deferral are effective primitives for understanding streaming social tasks. Code is publicly available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。