通过高效扩展词汇表提升专业文本摘要质量,训练更快更省参数。
Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization

- 在预训练分词器中加入领域专属词,只替换低效词汇以控制参数增长。
- 在法律与医学摘要任务中,生成摘要的语义相似度显著提升,新词使用更准确。
- 相比持续预训练快35%-55%,参数量减少最多达37%,适合资源受限场景。
通用领域预训练的大语言模型在应用到专业领域时常因分词效率低下导致性能下降。尽管持续预训练能缓解问题,但无法解决根本的词汇不匹配。为此,我们提出一种目标明确、参数高效的领域适配方法,将词汇适应与预训练结合用于基于LLM的文本摘要。该统一框架在预训练分词器中增补领域特定词元,同时选择性替换训练不足或不可达的词元,以限制参数增长。我们在Llama-3.1-8B和Qwen2.5-7B上评估该方法,在法律与医学摘要任务中采用面向专家的挑战性评估协议,这些任务通常包含大量过度碎片化的未登录词(OOV)。词汇适应算法通过提升生成摘要与参考摘要之间的语义相似度,整体改善了摘要质量。此外,适应后的模型生成的摘要包含更多恰当的新词和领域专用词,从而提升连贯性、相关性和忠实度。我们进一步观察到,该方法相较持续预训练可减少35%-55%的训练时间,并在参数量上比仅扩展方法减少最多37%。代码已公开于https://github.com/gb-kgp/VocabReplace-Then-Expand。
原文摘要 · Abstract (English)
Large language models pretrained on general-domain corpora often exhibit tokenization inefficiencies when applied to specialized domains. Although continual pretraining for domain adaptation partially alleviate performance degradation, it does not resolve the fundamental vocabulary mismatch. To address this gap, we introduce a targeted parameter-efficient domain adaptation approach that combines vocabulary adaptation with pretraining for LLM-based text summarization. Our unified framework augments pretrained tokenizers with domain-specific tokens while selectively replacing under-trained and unreachable tokens to limit parameter growth. We evaluate our approach on Llama-3.1-8B and Qwen2.5-7B across legal and medical summarization tasks on a challenge-oriented evaluation protocol focused on expert-driven text and summaries which typically has higher concentration of over-fragmented Out-of-Vocabulary (OOV) words. The vocabulary adaptation algorithm enhances the overall quality of the summarization model by improving semantic similarity between the generated summaries and their references. In addition, the adapted model produces summaries that incorporate more appropriate novel and domain-specific words, leading to improved coherence, relevance, and faithfulness. We further observe that our proposed approach significantly reduce training time by $35-55\%$ over continual pretraining and reduce parameter counts up to $37\%$ w.r.t expansion-only methods. We make the codebase publicly available at https://github.com/gb-kgp/VocabReplace-Then-Expand.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。