一站式工具，让大模型微调数据处理可追踪、可审计。
CuratorKIT : Data Curation and Synthetic Data Generation for LLM Post-Training
- 整合数据清洗、生成、质检全流程，支持100+大模型接口
- 每条数据有完整溯源链，拒绝样本带结构化原因
- 适配TRL/Unsloth等框架，适合需要可复现数据流水线的工程师
大语言模型后训练中的数据整理至关重要，但现有工具常将数据摄入、去重、合成生成和质量过滤分作独立阶段，导致难以追溯决策过程或理解样本被拒原因。CuratorKIT 是一个开源 Python 库，涵盖整个数据生命周期的可配置流水线。框架包含六种源格式读取器与自动模式识别，预生成阶段的数据卫生层可检测凭证、个人身份信息（PII）及有害内容，八项由大模型驱动的生成任务，三个互补的质量检查关卡，支持精确的幻觉验证、结构化自适应恢复，并提供五种兼容 TRL、Unsloth、AlignTune 的训练就绪导出格式。每个决策均记录于不可变的逐样本溯源链中，被拒样本附带结构化失败原因而非无声丢弃。CuratorKIT 通过 LiteLLM 支持 100 多个大模型服务提供商，提供 Python API 和 YAML 驱动的 CLI，专为需要在大规模下实现可复现、可审计数据流水线的实践者设计。
原文摘要 · Abstract (English)
Data curation is a critical part of post-training pipelines for large language models, yet existing tools often treat ingestion, deduplication, synthetic generation, and quality filtering as separate stages. This fragmentation makes it difficult to audit pipeline decisions or understand why individual samples are rejected. CuratorKIT is an open-source Python library that covers this full lifecycle in a single configurable pipeline. The framework is composed of six source format readers and automatic schema detection, a pre-generation data hygiene layer for credentials, PII, and toxic content, eight LLM-powered generation tasks, three complementary quality gates with provenance-exact hallucination verification, structured adaptive recovery, and five training-ready export formats compatible with TRL, Unsloth, and AlignTune. Every pipeline decision is recorded in an append-only per-sample provenance chain, and rejected samples carry structured failure reasons rather than being silently discarded. CuratorKIT supports 100+ LLM providers through LiteLLM, exposes both a Python API and a YAML-driven CLI, and is designed for practitioners who need reproducible, auditable data pipelines at scale .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。