无需高精度梯度，实现量化模型的高效训练。
ZOQO: Zero-Order Quantized Optimization
- 用零阶近似替代梯度计算，保持参数量化状态。
- 在大语言模型微调中性能接近全精度方法。
- 适合边缘设备等资源受限场景使用。
深度学习日益增长的计算与内存需求在资源受限环境中带来严峻挑战。我们提出一种零阶量化优化（ZOQO）方法，用于训练量化参数和运算的模型。该方法利用梯度符号的零阶近似，并自适应调整学习过程，无需进行全精度梯度计算即可维持参数量化。实验表明，尽管存在零阶与量化操作的限制，本方法在大型语言模型微调及黑盒对抗攻击任务中仍达到与全精度方法相当的性能，凸显其在低资源环境中的应用潜力。
原文摘要 · Abstract (English)
The increasing computational and memory demands in deep learning present significant challenges, especially in resource-constrained environments. We introduce a zero-order quantized optimization (ZOQO) method designed for training models with quantized parameters and operations. Our approach leverages zero-order approximations of the gradient sign and adapts the learning process to maintain the parameters' quantization without the need for full-precision gradient calculations. We demonstrate the effectiveness of ZOQO through experiments in fine-tuning of large language models and black-box adversarial attacks. Despite the limitations of zero-order and quantized operations training, our method achieves competitive performance compared to full-precision methods, highlighting its potential for low-resource environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。