用SQL在低配设备上跑大模型，速度提升明显
TranSQL+: Serving Large Language Models with SQL on Low-Resource Hardware
- 把大模型计算图转成纯SQL，在数据库里执行
- 在仅用CPU的环境下，预填充延迟降低20倍，解码速度提升4倍
- 适合资源受限场景，如无GPU的嵌入式设备
在内存有限、缺乏GPU的设备上部署大语言模型仍具挑战。本文提出TranSQL+，一种基于模板的代码生成器，将大模型计算图转化为可在关系型数据库中执行的纯SQL查询。该方法不依赖外部库，利用数据库成熟的向量化执行和外存处理能力实现高效推理。我们进一步提出行转列（ROW2COL）优化，提升矩阵运算中连接操作的效率。在Llama3-8B和DeepSeekMoE模型上评估显示，与DeepSpeed Inference和Llama.cpp相比，TranSQL+在低内存和仅CPU配置下，预填充延迟最多降低20倍，解码速度提升4倍。结果表明，关系型数据库是低资源硬件上部署大模型的可行环境。
原文摘要 · Abstract (English)
Deploying Large Language Models (LLMs) on resource-constrained devices remains challenging due to limited memory, lack of GPUs, and the complexity of existing runtimes. In this paper, we introduce TranSQL+, a template-based code generator that translates LLM computation graphs into pure SQL queries for execution in relational databases. Without relying on external libraries, TranSQL+, leverages mature database features, such as vectorized execution and out-of-core processing, for efficient inference. We further propose a row-to-column (ROW2COL) optimization that improves join efficiency in matrix operations. Evaluated on Llama3-8B and DeepSeekMoE models, TranSQL+ achieves up to 20x lower prefill latency and 4x higher decoding speed compared to DeepSpeed Inference and Llama.cpp in low-memory and CPU-only configurations. Our results highlight relational databases as a practical environment for LLMs on low-resource hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。