arXiv:2502.15054cs.LG2025-02KDD被引 11

Snapchat开源图神经网络框架GiGL,解决大规模图学习的工程挑战。

GiGL: Large-Scale Graph Neural Networks at Snapchat

  • 构建分布式图学习系统,整合数据预处理到模型部署全流程
  • 在两年内支持35个业务场景落地,涵盖好友与内容推荐
  • 兼容学术主流工具(如PyG),降低工业应用门槛

图神经网络(GNN)在机器学习领域的进展推动了其在大规模商业应用中的潜力。然而,工业界实际落地仍受限于大规模图学习带来的独特挑战。本文介绍Snapchat内部用于训练、推理与应用GNN的解决方案——GiGL(Gigantic Graph Learning),一个开源的大规模分布式图机器学习库。GiGL负责从关系型数据库中提取图数据、子图采样、分布式训练、推理及工作流编排。它可无缝对接学术界主流工具如PyTorch Geometric(PyG),专注于解决规模化和生产化难题,使内部工程师能聚焦模型设计。目前,GiGL已在多个生产环境使用,过去两年内支撑超过35项业务发布,覆盖好友推荐、内容推荐与广告等场景。本文详细阐述其架构设计、核心工具、扩展能力、跨领域案例研究及规模化应用的关键经验。GiGL已开源,地址为https://github.com/Snapchat/GiGL。

原文摘要 · Abstract (English)

Recent advances in graph machine learning (ML) with the introduction of Graph Neural Networks (GNNs) have led to a widespread interest in applying these approaches to business applications at scale. GNNs enable differentiable end-to-end (E2E) learning of model parameters given graph structure which enables optimization towards popular node, edge (link) and graph-level tasks. While the research innovation in new GNN layers and training strategies has been rapid, industrial adoption and utility of GNNs has lagged considerably due to the unique scale challenges that large-scale graph ML problems create. In this work, we share our approach to training, inference, and utilization of GNNs at Snapchat. To this end, we present GiGL (Gigantic Graph Learning), an open-source library to enable large-scale distributed graph ML to the benefit of researchers, ML engineers, and practitioners. We use GiGL internally at Snapchat to manage the heavy lifting of GNN workflows, including graph data preprocessing from relational DBs, subgraph sampling, distributed training, inference, and orchestration. GiGL is designed to interface cleanly with open-source GNN modeling libraries prominent in academia like PyTorch Geometric (PyG), while handling scaling and productionization challenges that make it easier for internal practitioners to focus on modeling. GiGL is used in multiple production settings, and has powered over 35 launches across multiple business domains in the last 2 years in the contexts of friend recommendation, content recommendation and advertising. This work details high-level design and tools the library provides, scaling properties, case studies in diverse business settings with industry-scale graphs, and several key lessons learned in employing graph ML at scale on large social data. GiGL is open-sourced at https://github.com/Snapchat/GiGL.

图神经网络分布式学习工业应用开源框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。