构建高效可增量更新的维基编辑历史数据架构，降低处理门槛。
Blocks Architecture (BloArk): Efficient, Cost-Effective, and Incremental Dataset Architecture for Wikipedia Revision History
- 将维基编辑历史从XML转为JSONL格式，提升并发与存储效率
- 支持增量修改已有数据仓库，减少重复计算与资源消耗
- 适合需要频繁更新维基数据的NLP研究者快速接入使用
维基百科修订历史（WikiRevHist）是自然语言处理的重要资源，但现有工具处理时需大量计算资源且难以复用。为此，我们提出高效的数据处理架构Blocks Architecture（BloArk），包含块（blocks）、段（segments）和仓库（warehouses）三部分，并设计构建器（builder）与修改器（modifier）核心流程。Builder将原始XML格式的修订历史转换为更高效的JSON Lines（JSONL）格式；Modifier则基于已建仓库实现增量修改，显著降低重复工作与资源开销。该架构可灵活扩展，适用于大规模维基修订历史处理及下游NLP任务。代码、文档与示例均已开源，采用GPL-2.0许可。
原文摘要 · Abstract (English)
Wikipedia (Wiki) is one of the most widely used and publicly available resources for natural language processing (NLP) applications. Wikipedia Revision History (WikiRevHist) shows the order in which edits were made to any Wiki page since its first modification. While the most up-to-date Wiki has been widely used as a training source, WikiRevHist can also be valuable resources for NLP applications. However, there are insufficient tools available to process WikiRevHist without having substantial computing resources, making additional customization, and spending extra time adapting others' works. Therefore, we report Blocks Architecture (BloArk), an efficiency-focused data processing architecture that reduces running time, computing resource requirements, and repeated works in processing WikiRevHist dataset. BloArk consists of three parts in its infrastructure: blocks, segments, and warehouses. On top of that, we build the core data processing pipeline: builder and modifier. The BloArk builder transforms the original WikiRevHist dataset from XML syntax into JSON Lines (JSONL) format for improving the concurrent and storage efficiency. The BloArk modifier takes previously-built warehouses to operate incremental modifications for improving the utilization of existing databases and reducing the cost of reusing others' works. In the end, BloArk can scale up easily in both processing Wikipedia Revision History and incrementally modifying existing dataset for downstream NLP use cases. The source code, documentations, and example usages are publicly available online and open-sourced under GPL-2.0 license.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。