打造可持续更新的开源语料库，让社区共建共享。
Dynaword: From One-shot to Continuously Developed Datasets
- 用社区协作框架实现语料库动态更新
- 丹麦版语料库数据量超同类四倍且完全开源
- 适合关注开放数据与可持续研究的团队
大规模语料库是自然语言处理研究与发展的基石。然而现有方法面临三大挑战：（1）依赖模糊授权来源，限制使用、共享与衍生作品；（2）静态发布模式阻碍社区参与，降低数据寿命；（3）质量保障仅限于发布团队，未利用社区智慧。为此，我们提出两种贡献：动态词汇（Dynaword）方法与丹麦动态词汇（Danish Dynaword）。Dynaword是一种支持持续更新的大规模开源语料库构建框架。丹麦动态词汇是其具体实现，验证了该方法并展示其潜力。该语料库包含超过四倍于同类发布版本的词元，完全开源，并获得产业界与学术界的多重贡献。仓库内置轻量级测试，确保数据格式、质量与文档规范，建立可持续的社区协作与数据演进机制。
原文摘要 · Abstract (English)
Large-scale datasets are foundational for research and development in natural language processing. However, current approaches face three key challenges: (1) reliance on ambiguously licensed sources restricting use, sharing, and derivative works; (2) static dataset releases that prevent community contributions and diminish longevity; and (3) quality assurance processes restricted to publishing teams rather than leveraging community expertise. To address these limitations, we introduce two contributions: the Dynaword approach and Danish Dynaword. The Dynaword approach is a framework for creating large-scale, open datasets that can be continuously updated through community collaboration. Danish Dynaword is a concrete implementation that validates this approach and demonstrates its potential. Danish Dynaword contains over four times as many tokens as comparable releases, is exclusively openly licensed, and has received multiple contributions across industry and research. The repository includes light-weight tests to ensure data formatting, quality, and documentation, establishing a sustainable framework for ongoing community contributions and dataset evolution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。