让大模型在浏览器里高效运行，无需服务器支持。
WebLLM: A High-Performance In-Browser LLM Inference Engine
- 用WebGPU和WebAssembly实现浏览器内高性能推理。
- 在同设备下性能可达本地原生的80%。
- 适合注重隐私与本地计算的网页应用开发者。
大型语言模型（LLMs）的进步带来了强大能力。尽管传统部署需服务器级GPU和云推理，但小型开源模型及日益强大的消费级设备使本地部署成为可能。浏览器作为平台具有普适性、天然的代理环境，并能屏蔽不同设备厂商的后端差异。为此，我们提出WebLLM，一个开源的JavaScript框架，可在浏览器中实现高性能的LLM推理。WebLLM提供类似OpenAI的API，便于集成到网页应用；利用WebGPU进行高效的本地GPU加速，通过WebAssembly实现高性能的CPU计算。结合机器学习编译器MLC-LLM与Apache TVM，WebLLM采用优化的WebGPU内核，克服了缺乏高性能WebGPU内核库的问题。评估显示，WebLLM在同设备下可保留高达80%的原生性能，仍有进一步缩小差距的空间。WebLLM为通用访问、隐私保护、个性化且由本地驱动的浏览器端大模型应用铺平了道路。代码已开源：https://github.com/mlc-ai/web-llm。
原文摘要 · Abstract (English)
Advancements in large language models (LLMs) have unlocked remarkable capabilities. While deploying these models typically requires server-grade GPUs and cloud-based inference, the recent emergence of smaller open-source models and increasingly powerful consumer devices have made on-device deployment practical. The web browser as a platform for on-device deployment is universally accessible, provides a natural agentic environment, and conveniently abstracts out the different backends from diverse device vendors. To address this opportunity, we introduce WebLLM, an open-source JavaScript framework that enables high-performance LLM inference entirely within web browsers. WebLLM provides an OpenAI-style API for seamless integration into web applications, and leverages WebGPU for efficient local GPU acceleration and WebAssembly for performant CPU computation. With machine learning compilers MLC-LLM and Apache TVM, WebLLM leverages optimized WebGPU kernels, overcoming the absence of performant WebGPU kernel libraries. Evaluations show that WebLLM can retain up to 80% native performance on the same device, with room to further close the gap. WebLLM paves the way for universally accessible, privacy-preserving, personalized, and locally powered LLM applications in web browsers. The code is available at: https://github.com/mlc-ai/web-llm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。