用低精度量化任务向量，大幅降低模型合并的内存占用。
Task Vector Quantization for Memory-Efficient Model Merging
- 将任务向量而非完整检查点进行量化，利用其权重范围窄的特点
- 2比特下仍保持性能，仅需全精度8%的内存
- 适合资源受限场景下的多任务模型高效合并
模型合并通过整合特定任务微调后的检查点，实现高效的多任务模型。然而，存储多个任务专属检查点需要大量内存，限制了可扩展性，也阻碍了在更大模型和多样任务上的应用。本文提出对任务向量（即预训练与微调检查点之差）进行量化，而非直接量化微调检查点。我们发现任务向量具有较窄的权重范围，可在现有任务向量合并框架中实现低精度量化（如4比特）。为进一步减少超低比特精度（如2比特）下的量化误差，我们引入残差任务向量量化，将任务向量分解为基向量与偏移分量，并按量化敏感度分配比特数，在有限内存预算下保持精度并最小化误差。图像分类与密集预测任务实验表明，该方法在仅使用全精度检查点8%内存的前提下，维持或提升了模型合并性能。
原文摘要 · Abstract (English)
Model merging enables efficient multi-task models by combining task-specific fine-tuned checkpoints. However, storing multiple task-specific checkpoints requires significant memory, limiting scalability and restricting model merging to larger models and diverse tasks. In this paper, we propose quantizing task vectors (i.e., the difference between pre-trained and fine-tuned checkpoints) instead of quantizing fine-tuned checkpoints. We observe that task vectors exhibit a narrow weight range, enabling low precision quantization (e.g., 4 bit) within existing task vector merging frameworks. To further mitigate quantization errors within ultra-low bit precision (e.g., 2 bit), we introduce Residual Task Vector Quantization, which decomposes the task vector into a base vector and offset component. We allocate bits based on quantization sensitivity, ensuring precision while minimizing error within a memory budget. Experiments on image classification and dense prediction show our method maintains or improves model merging performance while using only 8% of the memory required for full-precision checkpoints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。