用轻量模型动态调整缓存回复，提升聊天效率与相关性。
TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses
- 引入轻量级LLM动态修改缓存回复以匹配新请求。
- 在真实数据集上缓存利用率显著提升，质量接近顶尖模型。
- 适合高并发场景下需低成本低延迟的LLM服务部署。
大型语言模型每天处理数百万查询，高效响应缓存可有效降低计算成本与延迟。然而，由于对话的个性化特征及语义相似度搜索精度有限，保持缓存回复的相关性面临挑战。为此，我们提出TweakLLM——一种新型路由架构，利用轻量级LLM动态适配缓存回复以匹配当前输入。通过全面评估，包括用户侧比对、满意度投票及多智能体LLM辩论，结果表明TweakLLM在保持与前沿模型相当的回复质量的同时，显著提升缓存有效性。在真实世界数据集上的表现证明，该方案为高吞吐量LLM部署提供了可扩展、资源高效的缓存解决方案，且不损害用户体验。
原文摘要 · Abstract (English)
Large Language Models (LLMs) process millions of queries daily, making efficient response caching a compelling optimization for reducing cost and latency. However, preserving relevance to user queries using this approach proves difficult due to the personalized nature of chatbot interactions and the limited accuracy of semantic similarity search. To address this, we present TweakLLM, a novel routing architecture that employs a lightweight LLM to dynamically adapt cached responses to incoming prompts. Through comprehensive evaluation, including user studies with side-by-side comparisons, satisfaction voting, as well as multi-agent LLM debates, we demonstrate that TweakLLM maintains response quality comparable to frontier models while significantly improving cache effectiveness. Our results across real-world datasets highlight TweakLLM as a scalable, resource-efficient caching solution for high-volume LLM deployments without compromising user experience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。