ReSpec通过筛选相关且具体的视频文本数据,用5%的训练量实现顶尖检索效果。
ReSpec: Relevance and Specificity Grounded Online Filtering for Learning on Video-Text Data Streams
- 基于相关性与特异性动态过滤视频文本流,实时选择关键数据
- 仅用5%数据在5个零样本任务上达当前最佳性能,计算开销极低
- 适合需要实时响应的大规模视频理解场景,如在线推荐系统
视频-文本数据的快速增长带来了存储与计算挑战。在线学习通过实时处理数据流,提供高效解决方案,并支持实时响应场景。为提升学习效率与效果,我们提出基于相关性与特异性的在线过滤框架 ReSpec,依据四项标准筛选数据:(i) 模态对齐以确保数据干净,(ii) 任务相关性以聚焦目标任务,(iii) 特异性以捕捉信息丰富的细节内容,(iv) 效率以支持低延迟处理。相关性通过输入数据与下游任务的概率对齐度衡量,特异性则使用距离代表最不具体数据的根嵌入作为信息量的高效代理。通过目标任务数据建立参考点,ReSpec 实现数据的实时过滤,无需大量存储与计算。在 WebVid2M 与 VideoCC3M 等大规模数据集上评估,ReSpec 在五个零样本视频检索任务中达到当前最优表现,仅使用 5% 的数据量,且计算开销极小。源代码已公开于 https://github.com/cdjkim/ReSpec。
原文摘要 · Abstract (English)
The rapid growth of video-text data presents challenges in storage and computation during training. Online learning, which processes streaming data in real-time, offers a promising solution to these issues while also allowing swift adaptations in scenarios demanding real-time responsiveness. One strategy to enhance the efficiency and effectiveness of learning involves identifying and prioritizing data that enhances performance on target downstream tasks. We propose Relevance and Specificity-based online filtering framework (ReSpec) that selects data based on four criteria: (i) modality alignment for clean data, (ii) task relevance for target focused data, (iii) specificity for informative and detailed data, and (iv) efficiency for low-latency processing. Relevance is determined by the probabilistic alignment of incoming data with downstream tasks, while specificity employs the distance to a root embedding representing the least specific data as an efficient proxy for informativeness. By establishing reference points from target task data, ReSpec filters incoming data in real-time, eliminating the need for extensive storage and compute. Evaluating on large-scale datasets WebVid2M and VideoCC3M, ReSpec attains state-of-the-art performance on five zeroshot video retrieval tasks, using as little as 5% of the data while incurring minimal compute. The source code is available at https://github.com/cdjkim/ReSpec.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。