让生成式大模型学会词语的多种含义，提升知识蒸馏效果。
Decoder-based Sense Knowledge Distillation
- 将词义资源融入解码器训练，无需推理时查字典。
- 在多个基准上显著提升生成模型的知识蒸馏性能。
- 适合需要精准语义理解的文本生成任务使用。
大语言模型（LLMs）能捕捉丰富的上下文语义信息，但常忽略词义、词间关系等结构化词汇知识。已有研究证明，引入词义词典可提升编码器类模型的知识蒸馏效果，但对解码器类生成模型的应用仍具挑战。本文提出解码器型词义知识蒸馏（DSKD）框架，将词汇资源融入解码器训练过程，无需在推理阶段进行词典查询。大量实验表明，DSKD显著提升了解码器类模型的知识蒸馏表现，使生成模型在保持高效训练的同时继承结构化语义知识。
原文摘要 · Abstract (English)
Large language models (LLMs) learn contextual embeddings that capture rich semantic information, yet they often overlook structured lexical knowledge such as word senses and relationships. Prior work has shown that incorporating sense dictionaries can improve knowledge distillation for encoder models, but their application to decoder as generative models remains challenging. In this paper, we introduce Decoder-based Sense Knowledge Distillation (DSKD), a framework that integrates lexical resources into the training of decoder-style LLMs without requiring dictionary lookup at inference time. Extensive experiments on diverse benchmarks demonstrate that DSKD significantly enhances knowledge distillation performance for decoders, enabling generative models to inherit structured semantics while maintaining efficient training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。