arXiv:2609.01778cs.CV2026-09

动态分配视觉预算,提升长视频检索效果。

Allocate Before You Embed: Adaptive Visual Input Allocation for Video Embeddings

论文配图:Allocate Before You Embed: Adaptive Visual Input Allocation for Video Embeddings
图 1 · 摘自论文原文
  • 先分配后嵌入:用轻量预览器决定每帧分辨率,优化输入预算
  • 在固定预算下提升检索准确率,尤其保留原始分辨率时效果更优
  • 无需改动主模型,可适配多种视频检索系统

大规模视频检索需在有限的视觉输入和推理预算下编码长且多样的视频。现有方法通常以固定分辨率采样少量帧,限制了时间覆盖范围并忽略帧的重要性。我们的实证分析表明,在固定视觉输入预算下扩展时间覆盖范围能提升检索性能,且保留原始帧分辨率时增益更大,凸显时间覆盖与空间保真度的互补性。为此,我们提出 AllocEmbed 框架——先分配后嵌入,通过轻量级预览器在嵌入主干前重新分配固定视觉预算至更多帧。进一步引入基于检索反馈的策略优化(RDPO),利用排序验证的相似性差距与置信度引导的效率激励,直接学习分配策略。该框架完全在主干之前运行,无需修改嵌入模型或下游流程。在 MMEB-V2 V-QA 与 V-RET 任务及 LongRet 基准上的实验显示,AllocEmbed 在匹配预算的方法中表现最优,并具备跨主干迁移能力。代码已公开于 https://github.com/jinsong8/AllocEmbed。

原文摘要 · Abstract (English)

Large-scale video retrieval requires embedding models to encode long and diverse videos under tight visual-input and inference budgets. Existing methods typically sample a small, fixed set of frames at their original resolution, limiting temporal coverage and ignoring frame importance. Our empirical analysis shows that expanding temporal coverage improves retrieval even under a fixed visual-input budget. Gains are larger when the original per-frame resolution is preserved, highlighting the complementary roles of temporal coverage and spatial fidelity. Motivated by this finding, we propose AllocEmbed, an allocate-then-embed framework that reallocates a fixed visual-input budget across more frames. A lightweight allocator uses low-cost previews to assign frame-wise resolutions before the embedding backbone, preserving more detail where it most benefits retrieval while reducing visual cost elsewhere. We further introduce Retrieval-Driven Policy Optimization (RDPO), which learns the allocator directly from retrieval feedback using a rank-validated similarity gap and a confidence-guided efficiency incentive. Operating entirely before the backbone, AllocEmbed integrates with existing retrieval systems without modifying the embedding model or downstream pipeline. Experiments on the MMEB-V2 V-QA and V-RET tasks and our LongRet benchmark show that AllocEmbed achieves the best overall retrieval performance among the evaluated budget-matched methods and transfers across embedding backbones. Our code is publicly available at https://github.com/jinsong8/AllocEmbed.

视频检索自适应分配预算优化嵌入框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。