arXiv:2511.15512cs.CL2025-11

为语言数据处理建立标准化流程,提升可复现性。

Standardising the NLP Workflow: A Framework for Reproducible Linguistic Analysis

  • 设计类似脑成像标准的LPDS数据结构,规范文件组织
  • 开发pelican nlp工具包,一键执行从清洗到特征提取全流程
  • 配置文件驱动,适合需要高可复现性的语言研究者

大语言模型等AI技术的发展推动了语言数据分析方法的演进,但随之而来的是语言数据组织与共享缺乏标准,处理流程难以复现。为此,我们提出语言处理数据结构(LPDS),受神经科学领域广泛采用的脑成像数据结构(BIDS)启发,定义了统一的文件夹结构与命名规范。同时推出pelican nlp,一个模块化、可扩展的Python工具包,支持从数据清洗、任务特定预处理到语义嵌入、韵律度量等复杂特征提取的全流程自动化。整个处理流程可通过单一可共享的配置文件定义,pelican nlp在符合LPDS格式的数据上执行后,生成可复现的预处理数据或标准化的语言与声学特征及其聚合结果。LPDS与pelican nlp共同构建了端到端的语言数据处理流水线,旨在实现方法透明并增强研究可复现性。

原文摘要 · Abstract (English)

The introduction of large language models and other influential developments in AI-based language processing have led to an evolution in the methods available to quantitatively analyse language data. With the resultant growth of attention on language processing, significant challenges have emerged, including the lack of standardisation in organising and sharing linguistic data and the absence of standardised and reproducible processing methodologies. Striving for future standardisation, we first propose the Language Processing Data Structure (LPDS), a data structure inspired by the Brain Imaging Data Structure (BIDS), a widely adopted standard for handling neuroscience data. It provides a folder structure and file naming conventions for linguistic research. Second, we introduce pelican nlp, a modular and extensible Python package designed to enable streamlined language processing, from initial data cleaning and task-specific preprocessing to the extraction of sophisticated linguistic and acoustic features, such as semantic embeddings and prosodic metrics. The entire processing workflow can be specified within a single, shareable configuration file, which pelican nlp then executes on LPDS-formatted data. Depending on the specifications, the reproducible output can consist of preprocessed language data or standardised extraction of both linguistic and acoustic features and corresponding result aggregations. LPDS and pelican nlp collectively offer an end-to-end processing pipeline for linguistic data, designed to ensure methodological transparency and enhance reproducibility.

自然语言处理数据标准化可复现性Python工具

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。