LLMBridge降低大模型调用成本，让发展中国家学生也能用得起AI。
LLMBridge: Reducing Costs to Access LLMs in a Prompt-Centric Internet
- 用模型选型、上下文压缩和语义缓存三招降成本
- 部署12个月服务超100人，处理14700+请求，日均500次
- 适合教育场景和资源受限地区的低成本AI应用
当前互联网以HTTP内容检索为中心，中间件（如代理）在性能、安全和成本方面至关重要。我们设想未来通信将由发送给生成式AI模型的‘提示’主导。为此，需要具备类似HTTP代理功能（如缓存、路由、压缩）的智能代理，同时应对提示通信的独特挑战与机遇。作为第一步，我们提出LLMBridge——一个面向成本敏感用户（如发展中国家用户、教育场景中的师生）的LLM代理。该系统支持三项核心优化：模型选择（将提示路由至最合适的模型）、上下文管理（智能减少上下文长度）、语义缓存（使用本地模型和向量数据库响应提示）。这些优化在成本与质量间引入权衡，应用通过高层双向接口进行调节。我们在两个低预算场景中部署：一个基于WhatsApp的问答服务，已上线12个月，服务100+用户，处理超14,700次请求；另一个是大学课堂环境，覆盖三门计算机课程的学生，平均每天处理500次请求，支持推理代理、聊天机器人等多样化应用。我们报告了部署经验，并利用收集的工作负载对多种成本优化策略进行基准测试，分析其在成本、延迟和响应质量间的权衡。
原文摘要 · Abstract (English)
Today's Internet infrastructure is centered around content retrieval over HTTP, with middleboxes (e.g., HTTP proxies) playing a crucial role in performance, security, and cost-effectiveness. We envision a future where Internet communication will be dominated by "prompts" sent to generative AI models. For this, we will need proxies that provide similar functions to HTTP proxies (e.g., caching, routing, compression) while dealing with unique challenges and opportunities of prompt-based communication. As a first step toward supporting prompt-based communication, we present LLMBridge, an LLM proxy designed for cost-conscious users, such as those in developing regions and education (e.g., students, instructors). LLMBridge supports three key optimizations: model selection (routing prompts to the most suitable model), context management (intelligently reducing the amount of context), and semantic caching (serving prompts using local models and vector databases). These optimizations introduce trade-offs between cost and quality, which applications navigate through a high-level, bidirectional interface. As case studies, we deploy LLMBridge in two cost-sensitive settings: a WhatsApp-based Q&A service and a university classroom environment. The WhatsApp service has been live for over twelve months, serving 100+ users and handling more than 14.7K requests. In parallel, we exposed LLMBridge to students across three computer science courses over a semester, where it supported diverse LLM-powered applications - such as reasoning agents and chatbots - and handled an average of 500 requests per day. We report on deployment experiences across both settings and use the collected workloads to benchmark the effectiveness of various cost-optimization strategies, analyzing their trade-offs in cost, latency, and response quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。