小模型通过增强交叉注意力，从大模型学知识。
LLM Modules: Knowledge Transfer from a Large to a Small Model using Enhanced Cross-Attention
- 用增强交叉注意力连接大模型与小模型的表示
- 15轮训练后小模型表现接近蒸馏效果
- 适合算力有限但需高质量生成的场景
本文提出一种名为LLM Modules的架构，通过增强交叉注意力机制，实现从大型预训练模型向小型模型的知识迁移。在该方案中，将Qwen2-1.5B模型冻结，其特征表示经专门设计的注意力层传递至计算资源受限的GPT-Neo-125M模型进行训练。在Bespoke-Stratos-17k数据集上的实验表明，经过15轮训练后，组合模型生成的响应质量可与蒸馏方法相媲美。文章还讨论了模块化方法的优势，提供输入查询示例与对比分析，并展望了该方法的进一步扩展前景。
原文摘要 · Abstract (English)
In this work, we propose an architecture of LLM Modules that enables the transfer of knowledge from a large pre-trained model to a smaller model using an Enhanced Cross-Attention mechanism. In the proposed scheme, the Qwen2-1.5B model is frozen and its representations are passed through specially designed attention layers to the GPT-Neo-125M model, which is trained on limited computational resources. Experimental results on the Bespoke-Stratos-17k dataset demonstrate that after 15 epochs of training, the combined model generates responses comparable in quality to those obtained by distillation. We discuss the advantages of the modular approach, provide examples of input queries and comparative analysis, and outline prospects for further extension of the method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。