将视觉语言模型压缩为三值版本，适配边缘设备运行
Ternarization of Vision Language Models for use on edge devices
- 用k-means从预训练权重初始化，加速三值化过程
- 三值模型内存占用更低，生成速度最快，困惑度接近原模型
- 自定义三值矩阵乘法算子，支持在TensorFlow Lite上高效执行
我们提出一种将预训练视觉语言模型压缩为三值版本的方法，而非从零训练三值模型。基于k-means算法的新型初始化方案，可显著减少三值化时间。针对TensorFlow Lite引擎实现多种自定义算子以执行三值模型。在内存消耗、推理速度和困惑度方面，对比原模型及其二值与三值版本，发现使用自定义三值矩阵乘法算子的三值模型在内存占用与困惑度之间取得良好平衡，同时具备最快的词元生成速度。
原文摘要 · Abstract (English)
We propose a process to compress a pre-trained Vision Language Model into a ternary version of itself instead of training a ternary model from scratch. A new initialization scheme from pre-trained weights based on the k-means algorithm is proposed to reduce the ternarization time. We implement different custom operators for executing the ternary model on the TensorFlow Lite Engine. We compare the original model with its ternary and binary versions in terms of memory consumption, inference speed and perplexity. We find that the ternary model using our custom ternary matrix multiplication operator provides a good compromise in term of memory usage and perplexity, while having the fastest token generation speed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。