HOLA让大模型在边缘设备上高效运行，兼顾速度与准确率。
LLMs on a Budget? Say HOLA
- 采用分层推测解码加速推理，不损失模型质量。
- 在GSM8K和ARC测试中分别提升17.6%和10.5%准确率。
- 适合医疗、教育等需要实时响应的边缘部署场景。
在边缘设备上运行大语言模型受限于高计算与内存需求，阻碍了医疗、教育及嵌入式系统中的实时应用。现有方法如量化、剪枝和检索增强生成（RAG）仅提供部分优化，常牺牲速度或精度。本文提出端到端优化框架HOLA，内部采用分层推测解码（HSD）实现快速推理且无质量损失；外部通过AdaComp-RAG根据上下文动态调整检索复杂度。结合LoBi（融合结构化剪枝与量化），HOLA在边缘设备（如Jetson Nano）上显著提升性能：在GSM8K上取得17.6%的EMA提升，在ARC上实现10.5%的MCA增益，并降低延迟与内存占用，证明其可扩展且具备生产可用性。
原文摘要 · Abstract (English)
Running Large Language Models (LLMs) on edge devices is constrained by high compute and memory demands posing a barrier for real-time applications in sectors like healthcare, education, and embedded systems. Current solutions such as quantization, pruning, and retrieval-augmented generation (RAG) offer only partial optimizations and often compromise on speed or accuracy. We introduce HOLA, an end-to-end optimization framework for efficient LLM deployment. Internally, it leverages Hierarchical Speculative Decoding (HSD) for faster inference without quality loss. Externally, AdaComp-RAG adjusts retrieval complexity based on context needs. Together with LoBi, which blends structured pruning (LoRA) and quantization, HOLA delivers significant gains: 17.6% EMA on GSM8K, 10.5% MCA on ARC, and reduced latency and memory on edge devices like Jetson Nano--proving both scalable and production-ready.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。