提出可直接微调三值化大模型的新方法，无需解量化
Low-Rank Ternary Adaptation for Fine-Tuning Transformers
- 用低秩克罗内克分解表示三值权重的离散更新
- 在六组语言与视觉模型上恢复量化损失的大部分性能
- 适合需要极致压缩和高效微调的部署场景
三值化变换器具有极强的内存与计算效率，但现有基于低比特LoRA的方法无法直接微调三值权重。当前方法要么需解量化，将低比特基权重还原为高精度以合并适配权重，要么仅更新量化参数，导致合并后模型无法保持三值特性。本文提出三值乘法适配，通过低秩克罗内克分解将三值权重的离散更新（如符号翻转或置零）表示为两个小的三值矩阵，逐元素作用于原三值权重。该设计参数高效且表达能力强，保持三值域，支持无需解量化直接合并。在六种模型上的实验，包括三值化LLaMA-3 1B和3B及三值ViT-B/16，表明该方法恢复了量化带来的大部分性能损失，并优于强低比特与三值基线。代码已开源。
原文摘要 · Abstract (English)
Ternary transformers offer extreme memory and compute efficiency, but existing low-bit LoRA-based methods cannot directly fine-tune ternary weights. Current approaches either require dequantization, restoring low-bit base weights to higher precision to merge with adaptation weight, or update only quantization parameters, preventing a merged model that remains ternary. We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights such as sign flips or zeroing through a low-rank Kronecker factorization into two small ternary matrices applied element-wise to ternary weights. This design is parameter-efficient and expressive, preserves the ternary domain, and supports direct merging without dequantization. Experiments on six models across language and vision, including ternarized LLaMA-3 1B and 3B and a ternary ViT-B/16, demonstrate that our method recovers much of the performance lost to quantization and outperforms strong low-bit and ternary baselines. Code is available at https://github.com/alexmanoo/ternary_adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。