arXiv:2505.21242cs.CL2025-05ACL被引 9

医学文本摘要中,词汇适配能显著提升大模型在生僻词密集场景下的表现。

Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings

  • 通过词汇适配更新模型词表,缓解医疗术语的词外词问题。
  • 在高生僻词数据上,适配后模型摘要准确率提升约18%。
  • 医学专家评估认为适配后的摘要更相关、更忠实于原文。

大语言模型(LLMs)在医疗文本摘要任务中虽已取得显著进展,但现有研究多在整体数据集上报告性能,缺乏对高生僻词(OOV)或高新颖性数据点的细粒度评估。本研究发现,在此类困难场景下,模型性能显著下降。尽管Llama-3.1词表规模达约128K,仍存在医疗术语过度切分问题。我们提出并验证了多种词汇适配策略,结合两种持续预训练方法与三个基准医疗摘要数据集,系统评估其效果。实验表明,词汇适配可有效提升模型在复杂场景下的摘要质量。人工评估由医学专家完成,结果一致显示适配后摘要更具相关性与忠实度。代码库已公开于https://github.com/gb-kgp/LLM-MedicalSummarization-Benchmark。

原文摘要 · Abstract (English)

Large Language Models (LLMs) recently achieved great success in medical text summarization by simply using in-context learning. However, these recent efforts do not perform fine-grained evaluations under difficult settings where LLMs might fail. They typically report performance scores over the entire dataset. Through our benchmarking study, we show that LLMs show a significant performance drop for data points with high concentration of out-of-vocabulary (OOV) words or with high novelty. Vocabulary adaptation is an intuitive solution to this vocabulary mismatch issue where the LLM vocabulary gets updated with certain expert domain (here, medical) words or subwords. An interesting finding from our study is that Llama-3.1, even with a vocabulary size of around 128K tokens, still faces over-fragmentation issue with medical words. To that end, we show vocabulary adaptation helps improve the LLM summarization performance even in difficult settings. Through extensive experimentation of multiple vocabulary adaptation strategies, two continual pretraining strategies, and three benchmark medical summarization datasets, we gain valuable insights into the role of vocabulary adaptation strategies for customizing LLMs to the medical domain. We also performed a human evaluation study with medical experts where they found that vocabulary adaptation results in more relevant and faithful summaries. Our codebase is made publicly available at https://github.com/gb-kgp/LLM-MedicalSummarization-Benchmark.

医疗摘要词汇适配大模型生僻词

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。