测量词嵌入的内在维度，发现大模型存在冗余，训练初期维度快速下降。
Measuring Intrinsic Dimension of Token Embeddings
- 通过几何方法估算词嵌入的内在维度，揭示其实际分布空间远小于参数量
- 模型规模越大，嵌入空间冗余越严重，大模型中冗余率显著上升
- 训练早期内在维度急剧下降，为LoRA微调提供关键参考点
本研究测量了小规模语言模型和现代大语言模型中词嵌入的内在维度（ID），以量化其表示流形相对于外在维度的冗余程度。结果表明，嵌入空间通常位于比其外在维度更低的流形上；随着模型规模增大，冗余率持续上升；训练初期内在维度迅速下降。此外，在嵌入层应用LoRA时，困惑度在估计的内在维度附近出现突降，表明内在维度可作为LoRA适配的重要指导依据。
原文摘要 · Abstract (English)
In this study, we measure the Intrinsic Dimension (ID) of token embedding to estimate the intrinsic dimensions of the manifolds spanned by the representations, so as to evaluate their redundancy quantitatively compared to their extrinsic dimensionality. In detail, (1) we estimate the ID of token embeddings in small-scale language models and also modern large language models, finding that the embedding spaces often reside on lower-dimensional manifolds compared to their extrinsic dimensionality; (2) we measure the ID across various model sizes and observe an increase in redundancy rates as the model scale grows; (3) we measure the dynamics of IDs during the training process, and find a rapid ID drop in the early stages of training. Moreover, (4) when LoRA is applied to the embedding layers, we observe a sudden drop in perplexity around the estimated IDs, suggesting that the ID can serve as a useful guideline for LoRA application.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。