比较大模型不同学习方式对隐藏表示复杂度的影响
A Comparative Study of Learning Paradigms in Large Language Models via Intrinsic Dimension
- 用内在维度衡量模型表示空间的自由度
- 提示学习生成的表示维度高于微调
- 揭示提示学习更依赖上下文信息
大语言模型在自然语言任务上的表现可通过监督微调（SFT）和上下文学习（ICL）提升，二者机制不同：前者通过训练数据最小化损失更新参数，后者则在提示中嵌入任务示例而不改变参数。本研究利用内在维度（ID）分析这些学习范式对模型隐藏表示的影响。通过估计任务执行过程中表示之间的自由度，我们发现：随着微调进行，表示的内在维度逐渐变化；而上下文学习中演示数量越多，表示维度越高。进一步比较发现，无论任务如何，上下文学习产生的表示具有更高内在维度，说明其在嵌入空间中位于更高维流形上。
原文摘要 · Abstract (English)
The performance of Large Language Models (LLMs) on natural language tasks can be improved through both supervised fine-tuning (SFT) and in-context learning (ICL), which operate via distinct mechanisms. Supervised fine-tuning updates the model's weights by minimizing loss on training data, whereas in-context learning leverages task demonstrations embedded in the prompt, without changing the model's parameters. This study investigates the effects of these learning paradigms on the hidden representations of LLMs using Intrinsic Dimension (ID). We use ID to estimate the number of degrees of freedom between representations extracted from LLMs as they perform specific natural language tasks. We first explore how the ID of LLM representations evolves during SFT and how it varies due to the number of demonstrations in ICL. We then compare the IDs induced by SFT and ICL and find that ICL consistently induces a higher ID compared to SFT, suggesting that representations generated during ICL reside in higher dimensional manifolds in the embedding space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。