开源工具DPK让LLM数据准备轻松扩展到千核集群
Data-Prep-Kit: getting your data ready for LLM application development
- 模块化设计支持本地或大规模集群运行
- 已用于Granite模型数据处理，可灵活扩展新转换操作
- 适合需高效准备LLM训练数据的研究者和开发者
数据准备是大型语言模型（LLM）开发的第一步且至关重要。本文介绍了一个名为Data Prep Kit（DPK）的开源数据准备工具包，它易于使用、可扩展且具备灵活的规模适应能力。DPK架构支持用户根据需求在本地机器上进行数据准备，或无缝扩展至包含数千个CPU核心的集群。其内置一系列高可扩展性且可扩展的模块，用于转换自然语言与代码数据；若需额外转换，可通过丰富的支持机制轻松开发新模块。这些模块可独立使用或串联成流水线执行多步骤操作。本文阐述了DPK的架构，并展示了从小型到超大规模CPU集群下的性能表现。DPK模块已被用于Granite Models的数据准备。我们认为DPK为人工智能社区提供了宝贵工具，有助于提升LLM性能或通过检索增强生成（RAG）微调模型。
原文摘要 · Abstract (English)
Data preparation is the first and a very important step towards any Large Language Model (LLM) development. This paper introduces an easy-to-use, extensible, and scale-flexible open-source data preparation toolkit called Data Prep Kit (DPK). DPK is architected and designed to enable users to scale their data preparation to their needs. With DPK they can prepare data on a local machine or effortlessly scale to run on a cluster with thousands of CPU Cores. DPK comes with a highly scalable, yet extensible set of modules that transform natural language and code data. If the user needs additional transforms, they can be easily developed using extensive DPK support for transform creation. These modules can be used independently or pipelined to perform a series of operations. In this paper, we describe DPK architecture and show its performance from a small scale to a very large number of CPUs. The modules from DPK have been used for the preparation of Granite Models [1] [2]. We believe DPK is a valuable contribution to the AI community to easily prepare data to enhance the performance of their LLM models or to fine-tune models with Retrieval-Augmented Generation (RAG).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。