只用2-3个维度就能让BERT完成多数任务，证明其存在严重冗余。
Few Dimensions are Enough: Fine-tuning BERT with Selected Dimensions Revealed Its Redundant Nature
- 仅选取最终层2-3个维度即可达到良好效果。
- 除CLS外，其他输出向量信息等价，多数任务无需全维输入。
- 适合想压缩模型、提升效率的研究者或部署场景。
在GLUE任务上对BERT进行微调时，通常只选择最后一层输出的部分向量输入新全连接层。但哪些部分有效、各维度承载何种信息仍不明确。本研究系统评估了标记向量、层和维度的有效性与冗余性。结果表明：最后一层除CLS向量外的其余输出包含等效信息；多数任务仅需2-3个维度；低层贡献随任务下降，高层间差异小。冻结预训练层的影响及跨任务微调实验也显示，隐藏层在微调中可能发生显著变化，说明BERT存在显著冗余，具备同时处理多个任务的能力，其维度数量可能过多。
原文摘要 · Abstract (English)
When fine-tuning BERT models for specific tasks, it is common to select part of the final layer's output and input it into a newly created fully connected layer. However, it remains unclear which part of the final layer should be selected and what information each dimension of the layers holds. In this study, we comprehensively investigated the effectiveness and redundancy of token vectors, layers, and dimensions through BERT fine-tuning on GLUE tasks. The results showed that outputs other than the CLS vector in the final layer contain equivalent information, most tasks require only 2-3 dimensions, and while the contribution of lower layers decreases, there is little difference among higher layers. We also evaluated the impact of freezing pre-trained layers and conducted cross-fine-tuning, where fine-tuning is applied sequentially to different tasks. The findings suggest that hidden layers may change significantly during fine-tuning, BERT has considerable redundancy, enabling it to handle multiple tasks simultaneously, and its number of dimensions may be excessive.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。