根据文本提示动态选帧，提升视频检索效率与精度
Prompt-aware of Frame Sampling for Efficient Text-Video Retrieval
- 用文本提示指导轻量级模型选关键帧，避免盲目采样
- 在MSR-VTT数据集上实现49.0%的R@1准确率，延迟降低75.3%
- 适合部署在边缘设备上的高效视频检索系统开发者
在边缘设备上实现高效的文本-视频检索对实际应用至关重要。现有方法面临准确率与计算效率难以平衡的问题：均匀采样虽能覆盖内容但计算开销大，显著帧采样虽降低负担却因忽略查询而产生偏差。为此，我们提出ProCLIP框架，通过提示感知的帧采样策略，利用文本提示动态引导轻量级特征提取器选择语义相关帧，克服了传统显著帧采样依赖静态、不随查询变化的缺陷。同时采用两阶段候选剪枝策略：先用轻量模块快速粗筛，再以CLIP进行精细重排序，在保持准确率的同时大幅提升效率。跨基准测试显示，ProCLIP相比基线延迟降低75.3%，在MSR-VTT数据集上仍保持49.0%的R@1准确率。代码已公开于https://github.com/tiffylong/ProCLIP。
原文摘要 · Abstract (English)
Enabling efficient text-video retrieval on edge-end devices is critical for real-world applications. Yet, existing methods face a critical challenge in balancing accuracy and computational efficiency: uniform frame sampling methods ensure content coverage but incur prohibitive computational costs, while salient-frame sampling methods reduce overhead but suffer from query-agnostic frame selection that biases retrieval results. To address this, we propose ProCLIP, a user-centric framework that achieves state-of-the-art accuracy with significantly improved efficiency. We design a prompt-aware frame sampling strategy that dynamically guides lightweight feature extractors using textual prompts to select semantically relevant frames, overcoming the limitations of existing salient-frame sampling methods which rely on static, query-agnostic selection criteria. Moreover, we adopt a two-stage candidate pruning strategy that combines rapid coarse filtering via a lightweight module with CLIP-powered fine-grained re-ranking, enhancing retrieval efficiency while preserving accuracy. Experiments across benchmarks show ProCLIP achieves 75.3% latency reduction versus baselines while maintaining competitive accuracy, i.e., R@1=49.0 in MSR-VTT dataset. Code is available at https://github.com/tiffylong/ProCLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。