用大模型历史表示提升推荐系统知识迁移效率
LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation
- 将大模型中间表示作为用户历史输入给小模型，实现高效知识迁移
- 工业级数据上知识迁移率翻倍，转化率提升超1%
- 无需实时调用大模型，适合大规模推荐系统落地
知识蒸馏（KD）将大基础模型（FM）的单一预测结果传递给轻量级垂直模型（VM），但因仅传递标量信息，难以捕捉大模型学习到的丰富中间知识，导致知识迁移率持续下降。为突破此瓶颈，我们提出LoopFM（Learning frOm HistOrical RePresentations of FM），通过将大模型的中间嵌入表示结构化为下游小模型的输入特征（如用户历史序列），在不依赖实时大模型推理且无需模型架构耦合的情况下，构建高带宽知识传输通道。我们提供了理论框架，包含增益分解与迁移率分析。在三个公开基准上，LoopFM展现出显著的AUC提升（如淘宝广告数据集上提升6%+），并能与传统KD形成互补。在工业级系统（百亿样本、万亿参数大模型）中，相比纯KD，LoopFM将知识迁移率近似翻倍，在首次上线后半年内带来+0.5%转化率提升，后续两次独立上线分别实现+1.03%和+1.22%转化率增长。
原文摘要 · Abstract (English)
Knowledge distillation (KD) transfers a single scalar prediction from a large foundation model (FM) to compact vertical models (VMs), suffering from diminishing transfer ratio -- the fraction of FM improvement captured by the VM -- as a single scalar cannot convey the rich intermediate knowledge that larger FMs learn. To address this bottleneck, we propose LoopFM (Learning frOm HistOrical RePresentations of FM), a framework that opens a high-bandwidth transfer channel by structuring FM intermediate embeddings as input features (e.g., user history sequence) for downstream VMs, without requiring real-time FM inference at serving and architectural coupling between FM and VM. We provide a theoretical framework for LoopFM with a gain decomposition and transfer-ratio analysis. On three public benchmarks, LoopFM demonstrates strong AUC improvements (e.g., 6%+ on TaobaoAd) and complementary knowledge transfer capability with KD. On industrial-scale systems (billions of examples, trillion-parameter FMs), LoopFM approximately doubles the knowledge transfer ratio on top of KD, delivering a +0.5% conversion improvement in the first half after its initial launch, and +1.03% and +1.22% conversion improvement from two individual launches in the subsequent half.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。