为词语设计多义嵌入，让小模型高效模仿大模型的语义理解能力
Multi-Sense Embeddings for Language Models and Knowledge Distillation
- 用聚类法从大模型输出中提取词的不同含义向量
- 小模型通过模仿这些含义向量，在保持性能的同时大幅压缩体积
- 适合追求轻量化部署且需保留语义丰富性的场景
基于Transformer的大语言模型依赖上下文嵌入，对同一词在不同语境生成不同表示。然而，词语的实际含义（语义）数量有限。本文提出多义嵌入作为令牌的即插即用替代方案，以捕捉词语在语言中的多种用法。通过将大模型生成的嵌入进行聚类，并将聚类中心视为代表性语义嵌入，构建语义嵌入字典。此外，提出一种新型知识蒸馏方法，利用该语义字典训练更小的学生模型，使其模仿大模型的多义表达，实现显著的空间与推理时间节省，同时保持竞争性性能。在多个基准上的充分实验验证了该方法的有效性。代码已开源。
原文摘要 · Abstract (English)
Transformer-based large language models (LLMs) rely on contextual embeddings which generate different (continuous) representations for the same token depending on its surrounding context. Nonetheless, words and tokens typically have a limited number of senses (or meanings). We propose multi-sense embeddings as a drop-in replacement for each token in order to capture the range of their uses in a language. To construct a sense embedding dictionary, we apply a clustering algorithm to embeddings generated by an LLM and consider the cluster centers as representative sense embeddings. In addition, we propose a novel knowledge distillation method that leverages the sense dictionary to learn a smaller student model that mimics the senses from the much larger base LLM model, offering significant space and inference time savings, while maintaining competitive performance. Via thorough experiments on various benchmarks, we showcase the effectiveness of our sense embeddings and knowledge distillation approach. We share our code at https://github.com/Qitong-Wang/SenseDict
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。