一键生成高质量领域数据，让大模型轻松适配专业文档。
Easy Dataset: A Unified and Extensible Framework for Synthesizing LLM Fine-Tuning Data from Unstructured Documents
- 通过图形界面配置提取与分块策略，自动处理杂乱文档
- 用角色化提示生成多样问答对，提升数据多样性
- 支持人工审核中间结果，适合需要高精度数据的场景
大型语言模型在通用任务上表现优异，但将其适配到特定领域仍因高质量领域数据稀缺而困难。现有数据合成工具难以从异构文档中有效提取可靠微调数据。为此，我们提出 Easy Dataset，一个统一且可扩展的框架，通过直观的图形用户界面，将非结构化文档转化为可用于微调的数据。具体而言，Easy Dataset 允许用户轻松配置文本提取模型和分块策略，将原始文档转化为连贯文本片段；随后利用基于角色的提示方法，借助公开可用的大模型生成多样化的问答对。整个流程中，人机协同的可视化界面支持对中间输出的审查与优化，保障数据质量。在金融问答任务上的实验表明，使用合成数据微调后的模型显著提升了领域性能，同时保持了通用知识。源代码与可安装包已开源，GitHub 星标超过 9,000。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown impressive performance on general-purpose tasks, yet adapting them to specific domains remains challenging due to the scarcity of high-quality domain data. Existing data synthesis tools often struggle to extract reliable fine-tuning data from heterogeneous documents effectively. To address this limitation, we propose Easy Dataset, a unified framework for synthesizing fine-tuning data from unstructured documents via an intuitive graphical user interface (GUI). Specifically, Easy Dataset allows users to easily configure text extraction models and chunking strategies to transform raw documents into coherent text chunks. It then leverages a persona-driven prompting approach to generate diverse question-answer pairs using public-available LLMs. Throughout the pipeline, a human-in-the-loop visual interface facilitates the review and refinement of intermediate outputs to ensure data quality. Experiments on a financial question-answering task show that fine-tuning LLMs on the synthesized dataset significantly improves domain-specific performance while preserving general knowledge. The source code and installable package are available at https://github.com/ConardLi/easy-dataset and have garnered over 9,000 GitHub stars.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。