arXiv:2505.02060cs.CV2025-05中稿 · publication at the…

将人脸检测与追踪结合,实时生成带身份信息的视频故事

Transforming faces into video stories -- VideoFace2.0

  • 融合检测、识别与被动追踪实现高效人脸重识别
  • 实测帧率18-25fps,错误身份减少73%-93%
  • 适合影视制作与多模态数据集构建

人脸检测与识别自计算机视觉兴起之初便备受关注。受原始VideoFace数字设备启发,我们设计了可高效生成结构化视频故事(即基于身份的信息目录)的先进视频分析工具——VideoFace2.0。该系统用于输入视频中每个独特人脸的空间与时间定位,即人脸重识别(ReID),并支持其分类、特征提取及结构化输出,便于后续任务使用。该近实时解决方案专为电视制作、媒体分析场景设计,也可作为训练唇读与多模态语音识别等挑战性视觉任务机器学习模型所需大规模视频数据集的高效工具。实验验证了所提算法的有效性,其结合人脸检测、识别与被动跟踪机制,实现鲁棒高效的重识别。测试实现可在消费级笔记本上达到18-25帧/秒。消融实验表明,该算法在减少错误身份数量方面相对提升73%至93%。我们希望本工作与开源代码能推动类似专用视频分析工具的发展,降低未来高质量多模态数据集生产的门槛。

原文摘要 · Abstract (English)

Face detection and face recognition have been in the focus of vision community since the very beginnings. Inspired by the success of the original Videoface digitizer, a pioneering device that allowed users to capture video signals from any source, we have designed an advanced video analytics tool to efficiently create structured video stories, i.e. identity-based information catalogs. VideoFace2.0 is the name of the developed system for spatial and temporal localization of each unique face in the input video, i.e. face re-identification (ReID), which also allows their cataloging, characterization and creation of structured video outputs for later downstream tasks. Developed near real-time solution is primarily designed to be utilized in application scenarios involving TV production, media analysis, and as an efficient tool for creating large video datasets necessary for training machine learning (ML) models in challenging vision tasks such as lip reading and multimodal speech recognition. Conducted experiments confirm applicability of the proposed face ReID algorithm that is combining the concepts of face detection, face recognition and passive tracking-by-detection in order to achieve robust and efficient face ReID. The system is envisioned as a compact and modular extensions of the existing video production equipment. Presented results are based on test implementation that achieves between 18-25 fps on consumer type notebook. Ablation experiments also confirmed that the proposed algorithm brings relative gain in the reduction of number of false identities in the range of 73%-93%. We hope that the presented work and shared code implementation will stimulate further interest in development of similar, application specific video analysis tools, and lower the entry barrier for production of high-quality multi-modal datasets in the future.

人脸重识别视频分析多模态数据实时处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。