将大模型转为三值权重后,部分能力保留,适合做专项微调。
Capability-Stratified Degradation in Ternary Language Models
- 用7240万词元量化训练,将大模型转为{-1,0,+1}三值权重
- 关键事实能力下降至仅26.19%准确率,但10项任务仍保持77.1%性能
- 虽不适合作通用替代,但适合作特定任务的轻量基础模型
极端低比特推理为小型化模型和受限部署提供了路径。三值语言模型将权重限制在{-1,0,+1},逼近每个权重约1.585比特的理论极限。对于预训练模型,关键问题并非能否量化,而是哪些能力得以保留,以及是否仍适用于微调。我们通过7240万词元的量化感知训练(QAT),将Qwen3.5-0.8B(752M参数)转换为三值权重,得到模型Cloe。在29个基准、表示诊断和下游微调中评估其表现。结果表明降级非均匀:线性探针从全精度教师模型中恢复43.76%的MMLU答案,而从Cloe中仅恢复26.19%(接近随机水平),说明专精事实信息丢失;但Cloe在10个任务上仍保持77.1%的教师性能。重要的是,微调后其在SST-2上达到89.8%(教师为95.6%),在XSum上达79.4%教师保留率。降级归因于量化导致的信息损失及有限QAT预算下的恢复不足。我们还指出评估陷阱:标准答案字母评分失效(Cloe在98.6%的MMLU问题中只输出"A"),需采用延续生成评分。最终,三值转换不适合作通用替代,但仍是任务特异型模型的紧凑基底。
原文摘要 · Abstract (English)
Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to $\{-1,0,+1\}$, approaching the limit of $\log_2 3 \approx 1.585$ bits/weight. The practical question for a pretrained model is not simply whether weights can be quantised but which capabilities survive and whether it remains useful for adaptation. We explore this by converting Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT). The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning. The evidence shows non-uniform degradation. A linear probe recovers 43.76% of MMLU answers from the full-precision teacher's representations but only 26.19% from Cloe (near chance), indicating specialist factual information is lost. However, Cloe retains measurable performance on ten tasks, averaging 77.1% of teacher performance. Crucially, fine-tuning raises Cloe to 89.8% on SST-2 (95.6% of the matched teacher) and reaches 79.4% teacher retention on XSum. We attribute degradation to a combination of quantisation-induced information loss and incomplete recovery due to the limited QAT budget. We also highlight an evaluation pitfall: standard answer-letter scoring failed (Cloe emitted "A" on 98.6% of MMLU questions), necessitating continuation scoring. Ultimately, ternary conversion is unsuitable as a drop-in general replacement yet remains valuable as a compact substrate for task-specific models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。