构建云上AI平台,实现跨资源协同的GPU加速分析。
The AI_INFN Platform: Artificial Intelligence Development in the Cloud
- 基于Kubernetes与Virtual Kubelet实现GPU资源弹性调度。
- 支持LHC计算网格与CINECA等超算中心跨域协同运行。
- 适合需要跨平台部署ML工作流的科研团队使用。
机器学习正深刻改变研究人员创建、实施和运行数据密集型软件的方式。然而,其应用给计算基础设施带来显著挑战,尤其是在开发、测试和生产环境中协调硬件加速器访问方面。意大利国家核物理研究所(INFN)发起的AI_INFN项目旨在通过提供全面的技术支持,包括面向AI的计算资源访问,推动机器学习方法在各类研究场景中的应用。依托INFN云生态系统和云原生技术,项目强调加速器硬件的高效共享,同时保持研究所研究活动的多样性。本文描述了一个基于Kubernetes的平台部署与调试过程,该平台旨在简化基于GPU的数据分析工作流,并实现其在异构分布式资源上的可扩展执行。通过集成Virtual Kubelet和InterLink API的卸载机制,平台支持工作流跨越多个资源提供方,从全球大型强子对撞机计算网格站点到像CINECA Leonardo这样的高性能计算中心。我们将展示初步基准测试、功能测试及案例研究,验证平台在性能与集成方面的成果。
原文摘要 · Abstract (English)
Machine Learning (ML) is profoundly reshaping the way researchers create, implement, and operate data-intensive software. Its adoption, however, introduces notable challenges for computing infrastructures, particularly when it comes to coordinating access to hardware accelerators across development, testing, and production environments. The INFN initiative AI_INFN (Artificial Intelligence at INFN) seeks to promote the use of ML methods across various INFN research scenarios by offering comprehensive technical support, including access to AI-focused computational resources. Leveraging the INFN Cloud ecosystem and cloud-native technologies, the project emphasizes efficient sharing of accelerator hardware while maintaining the breadth of the Institute's research activities. This contribution describes the deployment and commissioning of a Kubernetes-based platform designed to simplify GPU-powered data analysis workflows and enable their scalable execution on heterogeneous distributed resources. By integrating offloading mechanisms through Virtual Kubelet and the InterLink API, the platform allows workflows to span multiple resource providers, from Worldwide LHC Computing Grid sites to high-performance computing centers like CINECA Leonardo. We will present preliminary benchmarks, functional tests, and case studies, demonstrating both performance and integration outcomes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。