追踪大规模模型训练的来源与过程，提升可复现性与能效
Provenance Tracking in Large-Scale Machine Learning Systems
- 构建yProv4ML库，按W3C标准收集训练过程的溯源数据
- 支持插件扩展，可集成到工作流系统中实现全流程追踪
- 适合关注模型可复现性与能源效率的研究者和工程师
随着大规模AI模型需求持续增长，如何在计算效率、执行时间、准确率和能耗之间取得平衡成为关键的多维度挑战。这不仅需要创新的算法与硬件架构，还需全面的监控、分析与理解工具来追踪模型训练与部署中的底层过程。数据与流程的溯源信息（provenance）已成为实现这一目标的核心要素。通过溯源数据，研究人员与工程师可洞察资源使用模式，发现效率瓶颈，并确保AI开发流程的可复现性与可问责性。为此，本文提出yProv4ML库，一种以JSON格式收集溯源数据的工具，符合W3C PROV与ProvML标准。该库强调灵活性与可扩展性，支持通过插件集成其他数据采集工具，并与yProv框架深度整合，可在工作流管理系统中实现任务级溯源配对。
原文摘要 · Abstract (English)
As the demand for large scale AI models continues to grow, the optimization of their training to balance computational efficiency, execution time, accuracy and energy consumption represents a critical multidimensional challenge. Achieving this balance requires not only innovative algorithmic techniques and hardware architectures but also comprehensive tools for monitoring, analyzing, and understanding the underlying processes involved in model training and deployment. Provenance data information about the origins, context, and transformations of data and processes has become a key component in this pursuit. By leveraging provenance, researchers and engineers can gain insights into resource usage patterns, identify inefficiencies, and ensure reproducibility and accountability in AI development workflows. For this reason, the question of how distributed resources can be optimally utilized to scale large AI models in an energy efficient manner is a fundamental one. To support this effort, we introduce the yProv4ML library, a tool designed to collect provenance data in JSON format, compliant with the W3C PROV and ProvML standards. yProv4ML focuses on flexibility and extensibility, and enables users to integrate additional data collection tools via plugins. The library is fully integrated with the yProv framework, allowing for higher level pairing in tasks run also through workflow management systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。