提出TeZO，通过时间维度低秩性降低大模型微调内存开销。
TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
- 将优化扰动建模为三维张量，用CP分解提取时间与模型的低秩结构。
- 在相同效果下，内存仅需MeZO-Adam的35%，训练成本显著降低。
- 适用于资源受限场景，适合追求高效微调的大模型研究者。
零阶优化（ZO）在大语言模型高效微调中展现出巨大潜力。近期工作引入梯度低秩性，提出低秩ZO估计器以减少GPU内存消耗。然而，现有方法仅关注单个梯度的低秩特性，忽略了所有梯度在整个训练过程中共享的共性——即所有梯度近似位于同一子空间。本文提出新型低秩ZO估计器TeZO，同时捕捉模型与时间维度上的低秩性。具体地，将ZO扰动沿时间维度表示为三维张量，采用经典多线性分解（CPD）提取每个低秩二维矩阵，大幅降低训练开销。TeZO可轻松扩展至Adam变体，内存消耗低于MeZO-SGD，且仅需MeZO-Adam约35%的内存。理论分析与大量实验验证其高效性，在保持接近最先进性能的同时，显著降低时间和内存开销。
原文摘要 · Abstract (English)
Zeroth-order optimization (ZO) has demonstrated remarkable promise in efficient fine-tuning tasks for Large Language Models (LLMs). In particular, recent advances incorporate the low-rankness of gradients, introducing low-rank ZO estimators to further reduce GPU memory consumption. However, most existing works focus solely on the low-rankness of each individual gradient, overlooking a broader property shared by all gradients throughout the training, i.e., all gradients approximately reside within a similar subspace. In this paper, we consider two factors together and propose a novel low-rank ZO estimator, TeZO, which captures the low-rankness across both the model and temporal dimension. Specifically, we represent ZO perturbations along the temporal dimension as a 3D tensor and employ Canonical Polyadic Decomposition (CPD) to extract each low-rank 2D matrix, significantly reducing the training cost. TeZO can also be easily extended to the Adam variant while consuming less memory than MeZO-SGD, and requiring about only 35% memory of MeZO-Adam. Both comprehensive theoretical analysis and extensive experimental research have validated its efficiency, achieving SOTA-comparable results with lower overhead of time and memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。