不用外部数据，让模型自己生成校准数据，提升压缩后性能。
Self-calibration for Language Model Quantization and Pruning
- 用模型自身生成合成校准数据，替代传统随机文本。
- 在多种模型和任务上表现稳定，常优于真实校准数据。
- 适合隐私敏感或训练数据不可公开的场景。
量化与剪枝是语言模型压缩的核心方法，可实现高效推理。现有后训练量化与剪枝方法依赖校准数据（少量无标签样本），通常从网络文本中随机采样，以反映训练数据分布。但该方式存在两大问题：(1) 校准数据不具代表性会损害模型性能；(2) 组织日益拒绝公开训练数据。本文提出自校准（self-calibration）方法，无需外部数据，仅利用模型自身生成合成校准数据，以更好逼近预训练数据分布。我们在多种模型、压缩方法与任务上对比了自校准与多个基线。结果表明，该方法在最大化下游任务性能方面持续具有竞争力，频繁超越使用真实数据的表现。
原文摘要 · Abstract (English)
Quantization and pruning are fundamental approaches for model compression, enabling efficient inference for language models. In a post-training setting, state-of-the-art quantization and pruning methods require calibration data, a small set of unlabeled examples. Conventionally, this is randomly sampled web text, aiming to reflect the model training data. However, this poses two key problems: (1) unrepresentative calibration examples can harm model performance, and (2) organizations increasingly avoid releasing model training data. In this paper, we propose self-calibration as a solution. Our approach requires no external data, instead leveraging the model itself to generate synthetic calibration data, with a view to better approximating the pre-training data distribution. We extensively compare the performance of self-calibration with several baselines, across a variety of models, compression methods, and tasks. Our approach proves consistently competitive in maximizing downstream task performance, frequently outperforming even using real data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。