构建可扩展的自动新词检测流程，从海量社交媒体文本中发现真实新词。
From 124 Million Tokens to 1,021 Neologisms: A Large-Scale Pipeline for Automatic Neologism Detection
- 结合语法规则与大模型分类，用双框架定义新词范围。
- 从12460万词元中筛选出1021个候选词，58.7%经人工验证为真新词。
- 代码与数据开源，适合语言学、NLP研究者使用。
我们提出一个可扩展、模块化的自动新词检测流程，结合规则过滤与大模型分类。该流程基于语法与非语法构词框架，共同界定新词范畴，并形成四类分类体系（新词、实体、外来词、无意义）。尽管架构设计具备模块化与可迁移性，本流程在2005至2024年间5.27亿条英语Reddit帖子上实现实例化。从中提取1.246亿个唯一词元，通过超过99.99%的降维，得到1021个新词候选，规模足以进行人工专家验证。多个大模型对每个候选词独立分类并以多数投票决定，最终验证发现模型间存在显著分歧，凸显大规模操作化新词检测的挑战。对全部1021个候选词的人工标注确认，其中599个（58.7%）为真实的词汇创新。流程代码、词汇编译脚本及标注候选列表已公开于https://github.com/DiegoRossini/neologism-pipeline。
原文摘要 · Abstract (English)
We present a scalable, modular pipeline for automatic neologism detection that combines rule-based filtering with LLM classification. The pipeline is grounded in two complementary word-formation frameworks, grammatical and extra-grammatical morphology, which jointly define the scope of what counts as a neologism and inform a four-class classification scheme (neologism, entity, foreign, none). While designed to be modular and transferable at the architectural level, the pipeline is instantiated on 527 million English-language Reddit posts spanning 2005-2024. From this corpus, we extract 124.6 million unique tokens and reduce them by over 99.99% to yield 1,021 neologism candidates, a set small enough for manual expert verification. Multiple LLMs independently classify each candidate via majority vote, with a final verification step, revealing substantial cross-model disagreement and highlighting the challenge of operationalizing neologism detection at scale. Manual annotation of all 1,021 candidates confirms that 599 (58.7%) are genuine lexical innovations. The pipeline code, vocabulary compilation scripts, and the annotated candidate list are available at https://github.com/DiegoRossini/neologism-pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。