发现大模型微调中存在大量冗余的词元更新，可删减30%-70%仍保持性能
TS-PEFT: Unveiling Token-Level Redundancy in Parameter-Efficient Fine-Tuning
- 用近端优化动态探测词元级冗余，智能识别哪些词元无需更新
- 删掉30%-70%词元更新后，性能仍不输甚至超过密集微调方法
- 揭示词元稀疏性比传统权重重要性更能反映模型适配机制
当前参数高效微调（PEFT）方法隐含假设：选定目标模块后，所有通过它的词元对下游任务贡献相同且均需参数更新。本文挑战这一假设，揭示大模型微调中普遍存在词元级冗余。提出TS-PEFT，一种基于近端优化的理论框架，可动态探测微调过程中的词元冗余。大量实验表明，盲目更新所有词元不仅计算浪费，还引入优化噪声。令人惊讶的是，舍弃30%-70%词元更新后，TS-PEFT性能始终匹配或超越密集基线（如LoRA、DoRA）。深入分析显示，学习到的词元级稀疏性比传统权重标准更优地指示模块重要性，为大模型内在适应机制提供了新的数据驱动视角。
原文摘要 · Abstract (English)
Current Parameter-Efficient Fine-Tuning (PEFT) methods typically operate under an implicit assumption: Once a target module is selected, every token passing through it contributes equally to the downstream task and requires a parameter update. In this paper, we challenge this convention by revealing a pervasive token-level redundancy in the fine-tuning of large models (LMs). We propose TS-PEFT, a theoretical framework utilizing proximal optimization that acts as a dynamic probe to identify token-level redundancy during the fine-tuning process. Extensive experiments demonstrate that indiscriminately updating all tokens is not only computationally superfluous but often introduces optimization noise. Surprisingly, by discarding 30%-70% of token updates, TS-PEFT consistently matches or exceeds the performance of dense baselines such as LoRA, DoRA. Our in-depth analysis shows that the learned token-level sparsity is a superior indicator of module importance compared to traditional weight criteria, providing a novel data-driven perspective on the intrinsic adaptation mechanism of LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。