用Python工具箱让大模型数据抽取更高效可复现。
DELM: a Python toolkit for Data Extraction with Language Models
- 模块化框架减少重复代码，支持批量处理与接口调用
- 内置验证和成本追踪，能量化评估不同抽取策略的性价比
- 适合需要反复实验、优化提示词或控制成本的研究者
大型语言模型（LLMs）已成为标注非结构化数据的强大工具。然而，现有工作大多依赖临时脚本，导致结果难以复现、系统不够鲁棒且评估不系统。为此，我们提出DELM（Data Extraction with Language Models），一个开源Python工具箱，旨在加速基于LLM的数据抽取流水线的实验迭代，并量化其间的权衡关系。DELM减少样板代码，提供模块化架构、结构化输出、内置验证、灵活的数据加载与评分策略，以及高效的批量处理能力。同时支持稳定调用LLM API，包含重试机制、结果缓存、详细的成本追踪和全面的配置管理。我们通过两个案例展示了其能力：一是新型提示优化算法，二是量化关键词选择在成本与覆盖范围之间的权衡。DELM已开源于github.com/Center-for-Applied-AI/delm。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have become powerful tools for annotating unstructured data. However, most existing workflows rely on ad hoc scripts, making reproducibility, robustness, and systematic evaluation difficult. To address these challenges, we introduce DELM (Data Extraction with Language Models), an open-source Python toolkit designed for rapid experimental iteration of LLM-based data extraction pipelines and for quantifying the trade-offs between them. DELM minimizes boilerplate code and offers a modular framework with structured outputs, built-in validation, flexible data-loading and scoring strategies, and efficient batch processing. It also includes robust support for working with LLM APIs, featuring retry logic, result caching, detailed cost tracking, and comprehensive configuration management. We showcase DELM's capabilities through two case studies: one featuring a novel prompt optimization algorithm, and another illustrating how DELM quantifies trade-offs between cost and coverage when selecting keywords to decide which paragraphs to pass to an LLM. DELM is available at \href{https://github.com/Center-for-Applied-AI/delm}{\texttt{github.com/Center-for-Applied-AI/delm}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。