arXiv:2606.12215cs.CVcs.IR2026-06KDD

通过多层级表示与时空匹配,高效去重海量近似视频。

MLT-Dedup: Efficient Large-Scale Online Video Deduplication via Multi-Level Representations and Spatial-Temporal Matching

论文配图:MLT-Dedup: Efficient Large-Scale Online Video Deduplication via Multi-Level Representations and Spatial-Temporal Matching
图 1 · 摘自论文原文
  • 用多级嵌入提取帧级与片段级特征,兼顾检索效率与精度。
  • 在线重复率降低91%,在90%精确率下实现高效去重。
  • 稀疏检索提升5倍索引容量,适合大规模平台部署。

在线平台用户生成视频内容爆炸式增长,伴随大量近似重复视频(仅部分编辑差异)的出现,这些重复视频降低用户体验并增加存储与带宽成本,使大规模视频去重成为关键任务。现有框架在有限索引预算下难以获取足够高质量候选视频,且效率与精度间存在权衡。为此,我们提出MLT-Dedup,一种基于多层级表示与时空匹配的高效大规模在线视频去重框架。其采用多层级视频编码器(ML-VE)提取细粒度帧级和稀疏片段级嵌入:稀疏嵌入用于高效候选检索,细粒度嵌入用于精确成对匹配。匹配阶段引入DiF-SiM(差分特征增强相似性模块),可定位重复时间片段并提供可靠相似性证据,支持策略驱动的去重决策。在真实大规模平台上的实验表明,MLT-Dedup在90%精确率下将在线重复率降低91%。此外,稀疏检索设计使索引容量提升5倍,显著扩大实际部署中的候选覆盖范围。

原文摘要 · Abstract (English)

The explosive growth of user-generated video content on online platforms is accompanied by the emergence of numerous near-duplicate videos--videos that are identical or highly similar but differ by partial edits. These duplicates degrade user experience and increase storage and bandwidth costs, making large-scale video deduplication a critical task. Existing video deduplication frameworks face a fundamental challenge in retrieving sufficient high-quality candidates under a limited index budget, as well as trade-offs between efficiency and precision. To address these issues, we propose MLT-Dedup, an efficient large-scale online video deduplication framework with Multi-Level representations and spatial-Temporal matching. Our approach employs a Multi-Level Video Encoder (ML-VE) to extract both fine-grained frame-level and sparse clip-level embeddings: sparse embeddings support efficient candidate retrieval, while fine-grained embeddings are loaded for precise pairwise matching. During matching, we introduce DiF-SiM, a Differential Feature-enhanced Similarity Module capable of locating duplicated temporal segments and providing reliable similarity evidence to support policy-driven deduplication decisions. Extensive experiments on a real-world large-scale platform demonstrate that MLT-Dedup reduces online repetition rates by 91% at 90% precision. Furthermore, our sparse retrieval design achieves a 5x increase in indexing capacity, enabling broader candidate coverage in real-world deployment.

视频去重多层级表示时空匹配大规模系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。