构建可实时检索的网页数据索引，提升大模型训练数据安全审查效率。
Going over Fine Web with a Fine-Tooth Comb: Technical Report of Indexing Fine Web for Problematic Content Search and Retrieval
- 基于ElasticSearch构建数据索引管道，支持快速查询。
- 在1.5TB多语言数据上实现毫秒级搜索响应，全查询<2秒。
- 为安全审查提供可落地工具，适合模型训练数据治理团队。
大型语言模型（LLMs）严重依赖如Common Crawl等网络规模数据集，其中超过80%的训练数据来自此类来源。然而，网络爬取的无差别特性带来了数据质量、安全与伦理问题。尽管训练数据质量至关重要，以往对有害内容的研究受限于计算能力，仅能分析小样本。本项目提出一个基于ElasticSearch的框架，用于索引与分析LLM训练数据集。我们将其应用于SwissAI的FineWeb-2语料库（1.5TB，涵盖四种语言），实现快速查询性能——多数搜索在毫秒级完成，所有查询均在2秒内。研究展示了实时数据集分析的可行性，为更安全、更可问责的人工智能系统提供了实用工具。
原文摘要 · Abstract (English)
Large language models (LLMs) rely heavily on web-scale datasets like Common Crawl, which provides over 80\% of training data for some modern models. However, the indiscriminate nature of web crawling raises challenges in data quality, safety, and ethics. Despite the critical importance of training data quality, prior research on harmful content has been limited to small samples due to computational constraints. This project presents a framework for indexing and analyzing LLM training datasets using an ElasticSearch-based pipeline. We apply it to SwissAI's FineWeb-2 corpus (1.5TB, four languages), achieving fast query performance--most searches in milliseconds, all under 2 seconds. Our work demonstrates real-time dataset analysis, offering practical tools for safer, more accountable AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。