arXiv:2603.18558cs.CVcs.AI2026-03被引 1

针对长视频问答中的帧选择难题,提出分层多模态选帧框架HiMu。

HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering

  • 将问题分解为分层逻辑树,逐原子谓词分配轻量级多模态专家
  • 在16帧预算下,多个基准测试达到当前最优准确率
  • 无需训练或多次调用大模型,可直接替换现有选帧模块

长视频问答需对长时间上下文进行推理,而帧选择是受有限上下文窗口限制的多模态大语言模型的关键瓶颈。现有方法在固定帧预算下仅基于单一全局查询嵌入评分,导致包含时序顺序或跨模态线索的复合问题(如“叙述提到反应后屏幕上发生了什么?”)的子事件顺序和模态绑定信息被丢失。我们提出 extbf{HiMu},一种无需训练的复合多模态帧选择框架。仅通过一次文本模型调用,将查询分解为分层逻辑树,其叶节点为原子谓词,分别路由至轻量级专家(视觉:CLIP、开放词汇检测、OCR;音频:语音识别与非语音声音匹配)。专家信号经归一化、跨模态平滑后,通过模糊逻辑算子自底向上组合,强制执行时序顺序与邻近性,生成连续的每帧满足度曲线。在Video-MME、LongVideoBench和HERBench-Lite的标准16帧预算下,HiMu在帧选择方法中达到最先进性能,作为即插即用模块,优于七种不同MLLM的均匀采样,且在四倍于均匀采样的帧数下达到相同准确率,无需重训练或多次迭代大模型调用。

原文摘要 · Abstract (English)

Long-form video question answering requires reasoning over extended temporal contexts, making frame selection a critical bottleneck for multi-modal large language models (MLLMs) bound by finite context windows. Within the controlled frame-budget regime that governs practical deployment, prior selectors score frames against a single global query embedding; as a result, compositional multimodal questions that involve temporal ordering or cross-modal cues such as ``what happens on screen right after the narrator mentions the reaction?'' are flattened into a representation that loses sub-event ordering and modality bindings. We introduce \textbf{HiMu}, a training-free framework for compositional multimodal frame selection. A single text-only LLM call decomposes the query into a hierarchical logic tree whose leaves are atomic predicates, each routed to a lightweight expert spanning vision (CLIP, open-vocabulary detection, OCR) and audio (speech recognition and non-speech sound matching). Expert signals are normalized, smoothed to align across modalities, and composed bottom-up through fuzzy-logic operators that enforce temporal sequencing and adjacency, yielding a continuous per-frame satisfaction curve. Under the standard 16-frame budget on Video-MME, LongVideoBench, and HERBench-Lite, HiMu achieves state-of-the-art accuracy among frame selection methods and improves over uniform sampling across seven diverse MLLMs as a drop-in module, matching the accuracy of uniform sampling at $4\times$ its frame budget, without retraining and without multiple iterative MLLM calls during selection.

视频问答帧选择多模态大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。