用冻结大模型提升短视频推荐,发现融合中间层特征更有效
Frozen LVLMs for Micro-Video Recommendation: A Systematic Study of Feature Extraction and Fusion
- 对比了替换与融合两种特征整合方式,发现融合更优
- 中间隐藏状态比生成的描述文本更能保留视觉细节,推荐效果更好
- 提出轻量级融合框架,可直接接入现有推荐系统
冻结的大规模视频语言模型(LVLM)在短视频推荐中应用日益广泛,因其具备强大的多模态理解能力。然而,其集成缺乏系统的实证评估:从业者通常将其作为固定黑箱特征提取器使用,未系统比较不同表征策略。为此,本文首次从两个关键设计维度展开系统性实证研究:(i) 与物品ID嵌入的整合策略,即替换与融合;(ii) 特征提取范式,对比LVLM生成的字幕与中间解码器隐藏状态。在代表性LVLM上的大量实验揭示三个核心原则:(1) 中间隐藏状态始终优于基于字幕的表征,因自然语言摘要不可避免地丢失对推荐至关重要的细粒度视觉语义;(2) ID嵌入捕获不可替代的协同信号,使融合严格优于替换;(3) 中间解码器特征的效果在不同层间差异显著。基于这些洞见,我们提出双特征融合(DFF)框架,一种轻量且即插即用的方法,能自适应融合来自冻结LVLM的多层表征与物品ID嵌入。DFF在两个真实世界短视频推荐基准上达到当前最优性能,持续超越强基线,为将现成大型视觉-语言模型融入短视频推荐系统提供了原则性方案。
原文摘要 · Abstract (English)
Frozen Large Video Language Models (LVLMs) are increasingly employed in micro-video recommendation due to their strong multimodal understanding. However, their integration lacks systematic empirical evaluation: practitioners typically deploy LVLMs as fixed black-box feature extractors without systematically comparing alternative representation strategies. To address this gap, we present the first systematic empirical study along two key design dimensions: (i) integration strategies with ID embeddings, specifically replacement versus fusion, and (ii) feature extraction paradigms, comparing LVLM-generated captions with intermediate decoder hidden states. Extensive experiments on representative LVLMs reveal three key principles: (1) intermediate hidden states consistently outperform caption-based representations, as natural-language summarization inevitably discards fine-grained visual semantics crucial for recommendation; (2) ID embeddings capture irreplaceable collaborative signals, rendering fusion strictly superior to replacement; and (3) the effectiveness of intermediate decoder features varies significantly across layers. Guided by these insights, we propose the Dual Feature Fusion (DFF) Framework, a lightweight and plug-and-play approach that adaptively fuses multi-layer representations from frozen LVLMs with item ID embeddings. DFF achieves state-of-the-art performance on two real-world micro-video recommendation benchmarks, consistently outperforming strong baselines and providing a principled approach to integrating off-the-shelf large vision-language models into micro-video recommender systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。