模型推理时动态量化，无需重训练即可适配新任务。
TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly
- 推理时在线校准激活值，动态调整量化参数。
- 在多个下游任务上实现比现有方法更快的推理速度。
- 适合部署在未知任务场景下的大模型加速应用。
为应对大基础模型带来的巨大计算需求，无需重训练的激活感知压缩技术被提出。然而，这些方法高度依赖校准数据，可能导致在未见下游任务中出现领域偏移问题。本文提出一种测试时量化（TTQ）框架，在推理时实时压缩大模型以解决该问题。通过高效在线校准，可即时根据每个提示自适应量化，无论下游任务如何，均能实现推理加速。多项实验表明，TTQ在量化性能上优于当前最优基线方法。
原文摘要 · Abstract (English)
To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these methods highly rely on calibration data, domain shift issues may arise for unseen downstream tasks. We propose a test-time quantization (TTQ) framework which compresses large models on the fly at inference time to resolve this issue. With an efficient online calibration, instant activation-aware quantization can adapt every prompt regardless of the downstream tasks, yet achieving inference speedup. Several experiments demonstrate that TTQ can improve the quantization performance over state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。