在开源RISC-V服务器上实现LLM推理加速，性能提升最高达3倍。
V-Seek: Accelerating LLM Reasoning on Open-hardware Server-class RISC-V Platforms
- 针对RISC-V架构优化大模型推理，适配多核向量处理器
- 在两款推理型大模型上实现4.32/2.29 token/s生成速度
- 适合关注国产硬件部署与低功耗推理的开发者
大型语言模型（LLMs）的迅猛发展依赖于基于GPU的系统。然而，对于推理与推理类工作负载，CPU正成为灵活且成本更低的替代方案，尤其在支持开放指令集架构（ISA）的RISC-V平台上。尽管如此，针对大模型任务的RISC-V硬件及其配套软件生态仍不够成熟，需领域特定调优。本文旨在填补这一空白，聚焦于在首个商用多核RISC-V CPU Sophon SG2042（具备向量处理能力）上优化大模型推理。针对两款面向推理优化的前沿大模型——DeepSeek R1 Distill Llama 8B和DeepSeek R1 Distill QWEN 14B，我们实现了4.32/2.29 token/s的生成速度以及6.54/3.68 token/s的提示处理速度，相较基线分别提速最高达2.9x和3.0x。
原文摘要 · Abstract (English)
The recent exponential growth of Large Language Models (LLMs) has relied on GPU-based systems. However, CPUs are emerging as a flexible and lower-cost alternative, especially when targeting inference and reasoning workloads. RISC-V is rapidly gaining traction in this area, given its open and vendor-neutral ISA. However, the RISC-V hardware for LLM workloads and the corresponding software ecosystem are not fully mature and streamlined, given the requirement of domain-specific tuning. This paper aims at filling this gap, focusing on optimizing LLM inference on the Sophon SG2042, the first commercially available many-core RISC-V CPU with vector processing capabilities. On two recent state-of-the-art LLMs optimized for reasoning, DeepSeek R1 Distill Llama 8B and DeepSeek R1 Distill QWEN 14B, we achieve 4.32/2.29 token/s for token generation and 6.54/3.68 token/s for prompt processing, with a speed up of up 2.9x/3.0x compared to our baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。