改进BPE分词策略，让模型更好理解专业领域术语。
Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models
- 先匹配新增专业词汇再分字符，提升分词准确性。
- 在分类和摘要任务中分别提升3.57%和1.87%准确率。
- 特别适合术语多或长文本的医学等领域，人类评估更满意。
本文揭示了现有基于字节对编码（BPE）的预训练语言模型微调方法在领域适配中的根本缺陷：简单将目标领域词汇追加至模型词表末尾，导致这些词汇优先级低，分词效果不佳。为此，我们提出AdaptBPE，其在BPE初始化阶段优先对新增词汇进行最长字符串匹配，再执行字符级分词。我们在多个分类与摘要任务上进行广泛评估，结果表明AdaptBPE在准确率上提升3.57%，在Rouge-L上提升1.87%。当参考摘要中未登录词（OOV）比例高或长度较长时，AdaptBPE（结合MEDVOC）表现尤为出色。人工评估也证实，AdaptBPE生成的摘要更相关、更忠实于原文。代码已开源：https://github.com/gb-kgp/adaptbpe。
原文摘要 · Abstract (English)
In this work, we show a fundamental limitation in vocabulary adaptation approaches that use Byte-Pair Encoding (BPE) tokenization scheme for fine-tuning pretrained language models (PLMs) to expert domains. Current approaches trivially append the target domain-specific vocabulary at the end of the PLM vocabulary. This approach leads to a lower priority score and causes sub-optimal tokenization in BPE that iteratively uses merge rules to tokenize a given text. To mitigate this issue, we propose AdaptBPE where the BPE tokenization initialization phase is modified to first perform the longest string matching on the added (target) vocabulary before tokenizing at the character level. We perform an extensive evaluation of AdaptBPE versus the standard BPE over various classification and summarization tasks; AdaptBPE improves by 3.57% (in terms of accuracy) and 1.87% (in terms of Rouge-L), respectively. AdaptBPE for MEDVOC works particularly well when reference summaries have high OOV concentration or are longer in length. We also conduct a human evaluation, revealing that AdaptBPE generates more relevant and more faithful summaries as compared to MEDVOC. We make our codebase publicly available at https://github.com/gb-kgp/adaptbpe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。