提出视频理解新压缩方法，提升长视频问答效率与精度。
EMCompress: Video-LLMs with Endomorphic Multimodal Compression
- 设计端态多模态压缩机制，保留关键语义同时减少冗余信息。
- 在VideoQA任务中，推理效率提升33.7%，训练效率提升7.33%。
- 适用于视频指令微调和问答系统，适合追求高效视频理解的研究者。
视频大模型在长视频推理中面临根本矛盾：静态稀疏采样要么过度包含无关片段而浪费资源，要么丢失精细时间语义。本文提出一种受认知启发的新任务——端态多模态压缩（EMC），将其建模为结构受限的充分统计量问题，并定义端态变换F_EMC : (V, Q) -> (v, q)，在保持答案不变的前提下压缩多模态输入。该形式使压缩输出直接处于下游任务空间，模仿认知文献中的“滤波-推理”机制，区别于传统隐变量压缩（IB/VIB），并可扩展至其他多模态场景。基于马尔可夫链A -> (V, Q) -> (v, q)，EMC实现了经典充分性条件I((v, q); A) = I((V, Q); A)的自然形式。作为模块化前端，EMC可集成至视频指令微调与视频问答流程。我们发布了首个专用基准，并提出ReSimplifyIt基线，其F-1提升0.40，且具备竞争力的查询重写能力。引入EMC后，视频语言理解在训练阶段相对增益7.33%，推理阶段达33.7%。
原文摘要 · Abstract (English)
Video-LLMs face a fundamental tension in long-video reasoning: static, sparse frame sampling either dilutes evidence across task-irrelevant segments at significant cost or misses fine-grained temporal semantics altogether. We propose a novel, cognitively-inspired task -- Endomorphic Multimodal Compression (EMC) -- as a structurally-constrained sufficient-statistic problem for VideoQA, and formulate it as an endomorphic transformation F_EMC : (V, Q) -> (v, q) that compresses the multimodal input while preserving answer invariance across reasonable downstream models. The endomorphic form keeps the compressed output in the downstream pipeline's native task space -- a structural mirror of the filter-then-reason mechanism in the cognitive literature motivating EMC -- distinguishing it from latent-code compression (IB / VIB) and making the formulation extensible to other multimodal settings. Under the Markov chain A -> (V, Q) -> (v, q), EMC realizes the classical sufficiency condition I((v, q); A) = I((V, Q); A) in its VideoQA-natural form. As a modular front-end, EMC plugs into both Video Instruction Tuning and Video Question Answering pipelines. We release the first dedicated benchmark and propose ReSimplifyIt, an EMC baseline surpassing prior methods by 0.40 F-1 with competitive query rewriting. Integrating EMC yields relative gains of 7.33% in training and 33.7% in inference for video-language understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。