用复数向量表示词元，实现小模型高精度文本分类
Wave Network: An Ultra-Small Language Model
- 用复数向量同时捕捉全局语义与词元间关系
- 仅240万参数模型在AG新闻上达91.66%准确率
- 比BERT小模型快85.6%且节省77.34%显存
我们提出一种新型超小型语言模型——Wave网络，采用复数向量表示每个词元，同时编码输入文本的全局与局部语义。复数向量由表征全局语义的模向量和捕获词元与全局语义关系的相位向量组成。在AG News文本分类任务上的实验表明，从随机初始化词嵌入生成复数向量时，单层Wave网络在波干扰下达到90.91%准确率，在波调制下达91.66%，分别优于使用BERT预训练嵌入的单层Transformer 19.23%和19.98%，并接近预训练微调的BERT base模型（94.64%）的性能。此外，相较于BERT base，Wave网络在波调制下显存占用降低77.34%，训练时间减少85.62%。结果表明，仅240万参数的Wave网络即可在文本分类任务中达到约1亿参数的BERT模型水平。
原文摘要 · Abstract (English)
We propose an innovative token representation and update method in a new ultra-small language model: the Wave network. Specifically, we use a complex vector to represent each token, encoding both global and local semantics of the input text. A complex vector consists of two components: a magnitude vector representing the global semantics of the input text, and a phase vector capturing the relationships between individual tokens and global semantics. Experiments on the AG News text classification task demonstrate that, when generating complex vectors from randomly initialized token embeddings, our single-layer Wave Network achieves 90.91% accuracy with wave interference and 91.66% with wave modulation - outperforming a single Transformer layer using BERT pre-trained embeddings by 19.23% and 19.98%, respectively, and approaching the accuracy of the pre-trained and fine-tuned BERT base model (94.64%). Additionally, compared to BERT base, the Wave Network reduces video memory usage and training time by 77.34% and 85.62% during wave modulation. In summary, we used a 2.4-million-parameter small language model to achieve accuracy comparable to a 100-million-parameter BERT model in text classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。