用视频大模型直接提取带世界知识的视觉表示,实现快速推荐。
LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendation
- 从原始帧中提取带语义的视觉标记,融合多层知识。
- 支持多视频历史输入,推理延迟低于传统方法。
- 无需人工标注,适合个性化推荐系统落地。
视频大语言模型(VLLMs)通过互联网规模数据预训练,具备世界知识感知能力,在电影分析和视频问答等任务中表现优异。然而,将其用于视频推荐仍面临挑战:真实系统需处理多视频输入、轻量级骨干网络、低延迟顺序推理及快速响应。现有方法存在三方面局限:(1) 仅解码生成导致推理延迟高;(2) 接口不支持多视频输入;(3) 输出限制为语言会丢失对下游视觉任务至关重要的细粒度视觉信息。我们指出,根源在于缺乏保留像素级细节同时利用世界知识的表示。为此提出LinkedOut,一种直接从视频中提取世界知识的表示方法,支持快速推理、多视频历史与去语言瓶颈。LinkedOut通过可提示查询和可选辅助模态,从原始帧中提取语义锚定的知识感知标记,并引入跨层知识融合MoE,自适应选择抽象层级,实现个性化、可解释、低延迟推荐。据我们所知,LinkedOut是首个基于VLLM、在原始帧上运行且无需人工标签的视频推荐方法,在标准基准上达到最先进性能。可解释性研究与消融实验验证了层多样性与层间融合的优势,为充分挖掘VLLM世界知识先验与视觉推理能力提供了可行路径。
原文摘要 · Abstract (English)
Video Large Language Models (VLLMs) unlock world-knowledge-aware video understanding through pretraining on internet-scale data and have already shown promise on tasks such as movie analysis and video question answering. However, deploying VLLMs for downstream tasks such as video recommendation remains challenging, since real systems require multi-video inputs, lightweight backbones, low-latency sequential inference, and rapid response. In practice, (1) decode-only generation yields high latency for sequential inference, (2) typical interfaces do not support multi-video inputs, and (3) constraining outputs to language discards fine-grained visual details that matter for downstream vision tasks. We argue that these limitations stem from the absence of a representation that preserves pixel-level detail while leveraging world knowledge. We present LinkedOut, a representation that extracts VLLM world knowledge directly from video to enable fast inference, supports multi-video histories, and removes the language bottleneck. LinkedOut extracts semantically grounded, knowledge-aware tokens from raw frames using VLLMs, guided by promptable queries and optional auxiliary modalities. We introduce a cross-layer knowledge fusion MoE that selects the appropriate level of abstraction from the rich VLLM features, enabling personalized, interpretable, and low-latency recommendation. To our knowledge, LinkedOut is the first VLLM-based video recommendation method that operates on raw frames without handcrafted labels, achieving state-of-the-art results on standard benchmarks. Interpretability studies and ablations confirm the benefits of layer diversity and layer-wise fusion, pointing to a practical path that fully leverages VLLM world-knowledge priors and visual reasoning for downstream vision tasks such as recommendation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。