用任务提示指导量化分配，让大模型更省内存还更准。
You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
- 通过隐藏层特征和输出敏感度判断每层重要性，动态分配精度。
- 在固定比特预算下，比通用量化方法提升准确率与内存效率。
- 无需训练、适合部署时优化模型压缩，尤其适合资源受限场景。
许多大模型应用只需特定功能，但传统后训练量化（PTQ）不考虑具体任务，导致部分层浪费比特而关键层过度压缩。本文提出任务感知量化（TAQ），一种无训练、仅修改权重的混合精度量化框架，利用少量未标注任务提示校准，将更高精度分配给任务相关层。TAQ基于隐藏表示和输出敏感度估计层重要性，设计三种评分规则：TAQ-IS（激活信息与稳定性）、TAQ-KL（量化噪声下的输出分布敏感度）、TAQ-O（标签引导的诊断基准）。在多个基准测试中，TAQ多数情况下优于任务无关基线，尤其在准确率-内存比上表现突出。硬件实测验证其吞吐量与延迟优势，并分析了校准鲁棒性与残差流误差传播。整体上，TAQ将混合精度量化从模型中心转向任务驱动。参考实现已开源。
原文摘要 · Abstract (English)
Many LLM applications require only narrow capabilities, yet standard post-training quantization (PTQ) methods allocate precision without considering the target task. This can waste bits on layers that are less relevant to the task signal while over-compressing layers that are critical for downstream behavior. We propose Task-Aware Quantization (TAQ), a training-free, weight-only mixed-precision PTQ framework that uses a small set of unlabeled task calibration prompts to allocate higher precision to task-relevant transformer layers under a fixed bit budget. TAQ estimates layer importance from hidden representations and output sensitivity, and we instantiate it with three scoring rules: TAQ-IS, based on activation information and stability; TAQ-KL, based on output-distribution sensitivity under a quantization-noise proxy; and TAQ-O, a label-informed oracle diagnostic for analyzing layer sensitivity. Across several benchmarks, TAQ outperforms task-agnostic baselines such in most settings, with especially strong gains in the accuracy--memory ratio. We further validate that these gains translate to real deployment behavior through hardware throughput and latency measurements, and analyze calibration robustness and residual-stream error propagation. Overall, TAQ turns mixed-precision PTQ from a model-centric compression step into a task-conditioned precision-allocation problem. A reference implementation is available at \href{https://anonymous.4open.science/r/TAQ-9217/README.md}{\includegraphics[height=1em]{imgs/github-mark.png}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。