轻量级GPT模型通过极简设计实现高效文本嵌入生成
Asterisk*: Keep it Simple
- 仅用两层、两注意力头、256维嵌入，架构极简
- 知识蒸馏后在分类任务上表现接近甚至超过更大模型
- 适合资源受限场景下的文本嵌入应用
本文介绍Asterisk，一种基于GPT的紧凑型文本嵌入生成模型。该模型采用极简架构，包含两层、两个注意力头和256维嵌入。通过从更大的预训练模型进行知识蒸馏，研究了模型规模与性能之间的权衡，同时最小化计算与内存开销。模型主要针对分类任务进行评估与优化，实验结果表明其在多种下游应用中的零样本分类任务中表现中等。通过额外配置，模型性能可在特定分类任务上达到或超越更大架构。
原文摘要 · Abstract (English)
This paper describes Asterisk, a compact GPT-based model for generating text embeddings. The model uses a minimalist architecture with two layers, two attention heads, and 256 embedding dimensions. By applying knowledge distillation from larger pretrained models, we explore the trade-offs between model size and performance while minimizing computational and memory requirements. The model is primarily evaluated and optimized for classification tasks, with experimental results showing its moderate performance in zero-shot classification across various downstream applications. With additional configuration, the model performance can approach or even surpass that of larger architectures on specific classification tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。