通过智能舍弃冗余令牌，大幅降低大模型微调内存占用。
TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditching
- 根据实例特征动态识别并舍弃低效令牌，实现内存优化。
- 在Llama3.2 1B模型上仅需14.8%的内存，性能持平或更优。
- 方法可解释性强，适用于多种Transformer模型，适合资源受限场景。
微调已成为将大语言模型（LLMs）适配下游任务的标准方法，但其继承自大模型的高训练内存开销导致效率低下。现有高效内存方案中，针对激活值的优化尤为有效，因激活值始终占总内存的主要部分。尽管已有多种激活优化策略，但其数据无关特性导致优化效果不佳且不稳定。本文提出TokenSeek，一种面向各类Transformer模型的通用插件式解决方案，通过实例感知的令牌搜索与舍弃机制，在显著降低微调内存消耗的同时保持甚至提升性能（例如在Llama3.2 1B模型上仅需14.8%内存）。此外，可解释的令牌选择过程揭示了其有效性背后的原理，为未来令牌效率研究提供重要启示。
原文摘要 · Abstract (English)
Fine tuning has been regarded as a de facto approach for adapting large language models (LLMs) to downstream tasks, but the high training memory consumption inherited from LLMs makes this process inefficient. Among existing memory efficient approaches, activation-related optimization has proven particularly effective, as activations consistently dominate overall memory consumption. Although prior arts offer various activation optimization strategies, their data-agnostic nature ultimately results in ineffective and unstable fine tuning. In this paper, we propose TokenSeek, a universal plugin solution for various transformer-based models through instance-aware token seeking and ditching, achieving significant fine-tuning memory savings (e.g., requiring only 14.8% of the memory on Llama3.2 1B) with on-par or even better performance. Furthermore, our interpretable token seeking process reveals the underlying reasons for its effectiveness, offering valuable insights for future research on token efficiency. Homepage: https://runjia.tech/iclr_tokenseek/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。