不同压缩方法在边缘设备上表现差异大，需根据任务选策略。
Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment
- 按任务选择压缩方式：问答用量化，分割用剪枝。
- 剪枝可能使树莓派延迟升3.4倍，量化反而稳定。
- 模型压缩会伪装出能力假象，需警惕结果误导。
在资源受限的边缘设备上运行大型AI模型需进行模型压缩以减小体积和计算量。然而，压缩效果好的模型未必部署效果好。本文综述数十项近期工作，基于真实硬件上的压缩结果提炼出实用部署指南。按照这些指南，我们在GPU、CPU及树莓派平台部署了紧凑型语言与图像模型，覆盖问答与图像分割任务。无单一技术在所有任务中占优：对于问答任务，Qwen3.5 0.8B在Q5_K_M GGUF量化下达到93.85 SQuAD F1和92 EM，而同精度下结构化剪枝损失16 F1，仅节省1%参数；对于分割任务，剪枝可减少近80%模型大小且保持近似不变的mIoU，而默认量化不改变参数与MACs。剪枝还可能因破坏k-quant超块对齐，导致部署产物膨胀21-49%，并伴随更长、格式不兼容的输出，使树莓派延迟最高达3.4倍。压缩甚至可能制造出虚假的模型能力：某LoRA恢复变体虽97/100预测集中于单类，仍保持71%严格BoolQ准确率，但平衡准确率仅52.6%。我们通过神经流图分析与预填充-解码级延迟分解解释这些现象，并归纳出任务导向的部署研究方向。最佳方案取决于任务、模型与硬件。实验代码与成果已开源。
原文摘要 · Abstract (English)
Running large AI models on resource-constrained edge devices requires model compression to reduce model size and computation. What compresses well, however, need not deploy well. We survey dozens of recent works that report compression results on real hardware and extract practical deployment guidelines from them. Following these guidelines, we deploy compact language and image models on GPU, CPU, and Raspberry Pi platforms across question answering and image segmentation. No single technique wins across tasks. For question answering, Qwen3.5 0.8B reaches 93.85 SQuAD F1 and 92 EM under Q5_K_M GGUF quantization, while structured pruning at the same precision costs 16 F1 at a 1% ratio. For segmentation, the ranking reverses: default quantization leaves parameters and MACs unchanged, whereas pruning cuts model size by nearly 80% at near-constant mIoU. Pruning can even inflate the deployed artifact by 21-49% by breaking k-quant super-block alignment; combined with longer, less format-compliant outputs, this raises Raspberry Pi latency up to 3.4x. Compression can also manufacture the appearance of competence rather than destroy it visibly: one LoRA-recovered variant stays fully parseable and holds 71% strict BoolQ accuracy while sending 97 of 100 predictions to a single class, at 52.6% balanced accuracy. We explain these effects through neural-flow graph analysis and prefill-decode-level latency decomposition, and condense them into task-specific deployment research directions. The right technique depends on the task, the model, and the hardware. Our experiment code and artifacts are open-sourced at https://github.com/Arnavvvkumar/deployment
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。