为制造业定制高效持续预训练模型，提升专业语义理解能力。
ManufactuBERT: Efficient Continual Pretraining for Manufacturing
- 基于制造业语料构建持续预训练框架，优化领域适应性。
- 在多个制造任务上超越现有基线，训练速度提升33%。
- 适合需要高效领域模型的工业文本分析场景。
通用Transformer编码器在制造业等专业领域表现不佳，因其缺乏领域术语和语义的充分训练。本文提出ManufactuBERT，一个在大规模制造业语料上持续预训练的RoBERTa模型。通过从网络数据中构建语料库，包含领域过滤与多阶段去重流程，有效消除冗余信息。实验表明，ManufactuBERT在多项制造相关NLP任务上达到新基准，优于现有专用模型。更重要的是，使用去重后语料可使收敛速度提升33%，显著降低训练时间和计算成本。该数据处理流程可复用于其他专业领域，实现高性能编码器的构建。模型与语料库将公开于https://huggingface.co/cea-list-ia。
原文摘要 · Abstract (English)
While large general-purpose Transformer-based encoders excel at general language understanding, their performance diminishes in specialized domains like manufacturing due to a lack of exposure to domain-specific terminology and semantics. In this paper, we address this gap by introducing ManufactuBERT, a RoBERTa model continually pretrained on a large-scale corpus curated for the manufacturing domain. We present a comprehensive data processing pipeline to create this corpus from web data, involving an initial domain-specific filtering step followed by a multi-stage deduplication process that removes redundancies. Our experiments show that ManufactuBERT establishes a new state-of-the-art on a range of manufacturing-related NLP tasks, outperforming strong specialized baselines. More importantly, we demonstrate that training on our carefully deduplicated corpus significantly accelerates convergence, leading to a 33\% reduction in training time and computational cost compared to training on the non-deduplicated dataset. The proposed pipeline offers a reproducible example for developing high-performing encoders in other specialized domains. We will release our model and curated corpus at https://huggingface.co/cea-list-ia.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。