构建了3.8TB高质量中文网页文本数据集，支持多维度细粒度信息标注。
ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information
- 设计工具链MDFG-tool，通过规则过滤噪声并自动评估质量、领域和毒性。
- 生成3.8TB中文文本数据，每条数据附带质量分、领域标签、毒性标签与得分。
- 适合需要可控、安全、领域特定训练数据的LLM研究人员使用。
在大语言模型（LLMs）发展过程中，预训练数据对模型能力塑造至关重要。近年来，已发布多个大规模高质量预训练数据集，如ChineseWebText1.0、C4、Pile、WanJuan、MAPCC等。然而，随着大模型持续演进，研究重点逐渐转向特定领域能力和安全性，此前粗粒度文本已难以满足训练需求。细粒度信息（如质量、领域、毒性）在构建可靠大模型方面日益重要。为此，本文提出一套名为MDFG-tool的新工具链，用于构建具有多维度和细粒度信息的大规模高质量中文数据集。首先，采用人工设计规则剔除原始内容中的明显噪声文本；其次，分别设计质量评估模型、领域分类器和毒性评估模型，对清洗后的数据进行评估；最后，将三类细粒度信息整合至每条文本。基于该方法，我们发布了目前最大、最高质量且具备细粒度标注的中文文本数据集ChineseWebText2.0，总容量达3.8TB，每条文本均配有质量分数、领域标签、毒性标签及毒性得分，便于研究人员根据具体需求筛选数据。数据、代码与工具链已在GitHub开源：https://github.com/CASIA-LM/ChineseWebText-2.0
原文摘要 · Abstract (English)
During the development of large language models (LLMs), pre-training data play a critical role in shaping LLMs' capabilities. In recent years several large-scale and high-quality pre-training datasets have been released to accelerate the research of LLMs, including ChineseWebText1.0, C4, Pile, WanJuan, MAPCC and others. However, as LLMs continue to evolve, focus has increasingly shifted to domain-specific capabilities and safety concerns, making those previous coarse-grained texts insufficient for meeting training requirements. Furthermore, fine-grained information, such as quality, domain and toxicity, is becoming increasingly important in building powerful and reliable LLMs for various scenarios. To address these challenges, in this paper we propose a new tool-chain called MDFG-tool for constructing large-scale and high-quality Chinese datasets with multi-dimensional and fine-grained information. First, we employ manually crafted rules to discard explicit noisy texts from raw contents. Second, the quality evaluation model, domain classifier, and toxicity evaluation model are well-designed to assess the remaining cleaned data respectively. Finally, we integrate these three types of fine-grained information for each text. With this approach, we release the largest, high-quality and fine-grained Chinese text ChineseWebText2.0, which consists of 3.8TB and each text is associated with a quality score, domain labels, a toxicity label and a toxicity score, facilitating the LLM researchers to select data based on various types of fine-grained information. The data, codes and the tool-chain are available on this website https://github.com/CASIA-LM/ChineseWebText-2.0
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。