用数据编程:让大模型自我改进,修复失败有据可依。
Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora

- 以结构化知识为桥梁,把数据工程变成可追踪的开发流程。
- 模型出错时能定位到具体知识缺失或推理断点并精准修复。
- 跨16个领域验证,适合想可靠训练专业模型的研究者。
将人类专业知识从文本可靠地迁移到大语言模型仍是人工智能的核心挑战。虽然在领域语料上微调已带来显著能力提升,但该过程缺乏反馈机制:当模型在特定任务上失败时,无法诊断训练数据的缺陷,只能盲目添加更多数据。本文提出,当从源语料中提取的结构化知识作为训练数据与评估的共同基础时,完整的数据工程生命周期可精确映射到软件开发生命周期:训练数据成为定义模型应学习内容的“源代码”,模型训练相当于“编译”,基准测试即“单元测试”,而基于失败的数据修复则等同于“调试”。在此框架下,模型失败可分解为概念级缺口和推理链断裂,能追溯至数据中的具体缺陷,并通过针对性修补修复,每次修复循环均能在不同模型规模和架构上实现一致改进,且不损害通用能力。我们形式化这一原则为“数据编程”,并在自然科学、工程、生物医学与社会科学共16个领域中实现,发布结构化知识库、基准套件和训练语料作为开放资源。本工作证明了训练数据与模型行为间的关系具有结构性可追溯性和系统性可修复性,为将人类专长可靠工程化进语言模型奠定了原则基础。
原文摘要 · Abstract (English)
Reliably transferring specialized human knowledge from text into large language models remains a fundamental challenge in artificial intelligence. Fine-tuning on domain corpora has enabled substantial capability gains, but the process operates without feedback: when a model fails on a domain task, there is no method to diagnose what is deficient in the training data, and the only recourse is to add more data indiscriminately. Here we show that when a structured knowledge representation extracted from the source corpus serves as the shared foundation for both training data and evaluation, the complete data-engineering lifecycle maps onto the software development lifecycle in a precise and operative way: training data becomes source code specifying what the model should learn, model training becomes compilation, benchmarking becomes unit testing, and failure-driven data repair becomes debugging. Under this correspondence, model failures decompose into concept-level gaps and reasoning-chain breaks that can be traced back to specific deficiencies in the data and repaired through targeted patches, with each repair cycle producing consistent improvements across model scales and architectures without degrading general capabilities. We formalize this principle as Programming with Data and instantiate it across sixteen disciplines spanning the natural sciences, engineering, biomedicine, and the social sciences, releasing a structured knowledge base, benchmark suite, and training corpus as open resources. By demonstrating that the relationship between training data and model behaviour is structurally traceable and systematically repairable, this work establishes a principled foundation for the reliable engineering of human expertise into language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。