用傅里叶变换生成嵌入向量，大幅减少参数量且性能不降。
Parameter-Efficient Transformer Embeddings
- 通过傅里叶展开直接生成嵌入，替代传统查表方式。
- 参数量减少显著，零样本下在STS-B上表现接近基准模型。
- 无需丢弃层，训练更快，适合资源受限场景。
基于Transformer的NLP模型中，嵌入层通常占据最大参数量，其规模随词表增大而增长，但性能提升不成比例。本文提出一种新方法：先通过归一化令牌ID的傅里叶展开确定性生成嵌入向量，再经轻量MLP捕捉高阶交互。我们在自然语言推理任务（SNLI和MNLI）上训练标准Transformer与本架构，并在句子文本相似性（STS-B）上评估零样本性能。结果表明，该方法在显著减少参数量的同时保持竞争力，训练速度更快，且无需使用丢弃（dropout）。此概念验证研究展示了可扩展、内存高效的语言模型潜力，激励后续大规模实验。
原文摘要 · Abstract (English)
Embedding layers in transformer-based NLP models typically account for the largest share of model parameters, scaling with vocabulary size but not yielding performance gains proportional to scale. We propose an alternative approach in which token embedding vectors are first generated deterministically, directly from the token IDs using a Fourier expansion of their normalized values, followed by a lightweight multilayer perceptron (MLP) that captures higher-order interactions. We train standard transformers and our architecture on natural language inference tasks (SNLI and MNLI), and evaluate zero-shot performance on sentence textual similarity (STS-B). Our results demonstrate that the proposed method achieves competitive performance using significantly fewer parameters, trains faster, and operates effectively without the need for dropout. This proof-of-concept study highlights the potential for scalable, memory-efficient language models and motivates further large-scale experimentation based on our findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。