将40亿参数模型压缩至1.641有效比特，探索极低比特存储与推理的平衡。
Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment
- 采用旋转+三值量化+误差补偿，实现权重仅存1.641比特
- 十项任务准确率从64.5%降至54.7%，部分任务保留超80%性能
- 模型体积减半至3.96GiB，但推理速度未提升，适合存储敏感场景
超低比特语言模型可降低存储与内存带宽，但名义上的“1.58比特”无法全面反映存储表示、保留能力或运行行为。本文研究了基于KOTMS旋转、E2M-ATQ三值量化和GPTQ式误差补偿的端到端后训练转换方法，对指令微调的40亿参数模型Qwen进行压缩。实验仅量化权重，激活保持16位精度，省略ILA-AMP。评估涵盖有效比特核算、任务能力保留、困惑度、校准敏感性、检查点组成及部署行为。最终转换使用1.641有效比特/权重（81.62%参数被量化）。在十项评分任务中，准确率从64.5%降至54.7%：BoolQ保留84.6%教师性能，而ARC-Challenge仅剩43.8%。困惑度在WikiText-2上由13.639升至18.748，PTB由24.700升至31.992，C4由19.831升至28.966。后续打包操作保留三值结构与缩放系数，模型大小从8.29 GiB降至3.96 GiB，困惑度基本不变。第三方打包尝试损失严重，未纳入主结果。打包产物未完成端到端任务精度与生成吞吐量测试。初步Triton GEMV微基准测试显示，单个测试形状下比FP16 cuBLAS慢4.6倍，因此不主张压缩本身能带来更快推理。
原文摘要 · Abstract (English)
Ultra-low-bit language models can reduce storage and memory bandwidth, but a nominal "1.58-bit" label does not fully describe the stored representation, retained capability, or runtime behavior. We study an end-to-end post-training conversion of Qwen, an instruction-tuned 4B-parameter model, using KOTMS rotation, E2M-ATQ ternarization, and GPTQ-style error compensation from TWLA. The experiment is weight-only: activations remain at 16-bit precision, so ILA-AMP is omitted. We evaluate effective bit accounting, task capability retention, perplexity, calibration sensitivity, checkpoint composition, and deployment behavior. The final conversion uses 1.641 effective bits per weight for quantized linear weights, with 81.62% of model parameters targeted. Across ten scored capability comparisons, accuracy falls from 64.5% to 54.7%. Degradation is uneven: BoolQ retains 84.6% chance-corrected teacher performance, while ARC-Challenge retains 43.8%. Perplexity rises from 13.639 to 18.748 on WikiText-2, 24.700 to 31.992 on PTB, and 19.831 to 28.966 on C4. A subsequent packing run preserves the ternary planes and scales, reducing reported model size from 8.29 GiB to 3.96 GiB with essentially unchanged perplexity. A separate third-party packing attempt was lossy and is excluded from the primary artifact claim. The packed artifact has not been benchmarked end-to-end for task accuracy or generation throughput. A preliminary Triton GEMV microbenchmark is 4.6x slower than FP16 cuBLAS on one tested shape. We therefore do not claim that compression alone yields faster inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。