Netflix用多模态嵌入实现个性化内容推荐,提升冷启动效果和推荐精度。
Multimedia Asset Personalization via Multimodal Embeddings at Netflix

- 用CLIP图像嵌入实现跨标题、跨画布的通用模型,替代五个独立模型。
- 三模态模型MediaFM在视频预览推荐中优于纯视觉基线,线上测试表现更优。
- 通过离线预测热度胜者加速模型筛选,降低实验成本,适合工程落地。
个性化推广素材(如封面图和视频预览片段)对Netflix的内容发现至关重要。传统基于ID的交互历史模型无法感知素材内容,难以支持新上线内容。我们介绍多模态嵌入如何重塑Netflix生产系统,并为采用基础模型嵌入的推荐系统提供可复用经验。首先,预训练图像嵌入实现跨标题、跨画布的知识迁移。将CLIP图像嵌入引入双塔模型后,单一模型可服务五种不同画布类型,取代五个独立训练的模型,显著改善冷启动性能。轻量级扩展利用CLIP的联合文本-图像空间,使封面推荐具备查询感知能力。其次,多模态显著优于单一模态。我们提出自研三模态基础模型MediaFM,基于大规模电视剧片段数据集,融合视觉(SeqCLIP)、音频(wav2vec 2.0)和时间文本信号;应用于视频预览个性化推荐,其离线与在线A/B测试表现均优于强基线。第三,设计一个简单的离线代理任务——仅凭嵌入预测热度胜者,能有效排序嵌入模型版本,提前筛选候选,大幅缩短实验与产品化周期;该任务现作为每个新MediaFM检查点的准入门槛。我们还分享了生产部署的关键工程决策(共享嵌入基础设施、低延迟服务、低成本筛选),以及设计权衡与失败模式。
原文摘要 · Abstract (English)
Personalized promotional assets, namely artwork images and video preview clips, are critical to content discovery on Netflix. Traditional models for asset selection rely on ID-based interaction history, leaving them blind to asset content and unable to serve newly launched titles and assets. We describe how multimodal embeddings reshaped production systems at Netflix and report transferable lessons for practitioners adopting foundation-model embeddings into recommender systems. First, pretrained image embeddings unlock cross-title, cross-canvas knowledge transfer. Augmenting a two-tower model with CLIP image embeddings lets a single model serve all five Netflix artwork canvas types, replacing five separately trained per-canvas models and substantially improving cold-start performance. A lightweight extension reuses CLIP's joint text-image space to make artwork personalization query-aware in search. Second, multimodality decisively beats any single modality for video preview personalization. We describe MediaFM, our in-house tri-modal foundation model trained on a large-scale corpus of shots from the Netflix show catalog, fusing visual (SeqCLIP), audio (wav2vec 2.0), and timed-text signals; adopted for video preview personalization, it outperforms strong visual-only baselines both offline and in online A/B tests. Third, a simple offline proxy task whose performance correlates with online outcomes can accelerate the experimentation and productization cycle. Predicting the popularity-based winner from embeddings alone ranks embedding models and versions, pruning the choice space before any end-to-end integration or A/B test; it now gates every new MediaFM checkpoint. We also share the production engineering decisions (shared embedding infrastructure, low-latency serving, cheap screening) that made these deployments viable, along with the design tradeoffs and failure modes we encountered.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。