改进Adam优化器，让大模型词向量更均衡
Better Embeddings with Coupled Adam
- 让Adam的动量与自适应学习率耦合，缓解词向量偏斜
- 在大规模数据上显著提升嵌入质量，下游任务准确率更高
- 适合追求高质量词向量的NLP研究者和工程师
尽管大语言模型表现出色，但其学到的词向量常呈现难以理解的各向异性特征。本文指出，Adam优化器中的二阶矩是导致该问题的原因，并提出一种名为耦合Adam（Coupled Adam）的改进优化器以缓解此现象。实验表明，耦合Adam能显著提升嵌入质量，在足够大的数据集上还带来了上游和下游任务性能的提升。
原文摘要 · Abstract (English)
Despite their remarkable capabilities, LLMs learn word representations that exhibit the undesirable yet poorly understood feature of anisotropy. In this paper, we argue that the second moment in Adam is a cause of anisotropic embeddings, and suggest a modified optimizer called Coupled Adam to mitigate the problem. Our experiments demonstrate that Coupled Adam significantly improves the quality of embeddings, while also leading to better upstream and downstream performance on large enough datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。