通过预计算表结构缓存，加速Text-to-SQL查询响应
TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQL
- 基于主外键关系预生成表的KV缓存，支持跨查询复用
- 在真实数据集上实现3.62倍TTFT提速，精度几乎无损失
- 适合需要低延迟的数据库问答系统部署
在Text-to-SQL任务中，现有基于大模型的方法常将完整数据库模式放入提示词，导致上下文过长、预填充延迟高。虽然用户查询通常涉及固定表集合，可实现KV缓存共享，但当前推理引擎（如SGLang和vLLM）在处理不同表序的查询时仍会重复生成前缀缓存。为此，我们提出离线预计算表表示为KV缓存，并在线按需查询。核心在于保持表间的主外键关系以保证语义一致性。此外，构建表Trie结构以高效支持推理阶段的缓存查找。为提升缓存性能，引入查询重排序策略以提高命中率，并设计并行化计算加载流水线，实现模型推理与缓存加载同步进行。实验表明，TableCache在保持极小性能下降的前提下，实现了最高达3.62倍的首次令牌时间（TTFT）加速。
原文摘要 · Abstract (English)
In Text-to-SQL tasks, existing LLM-based methods often include extensive database schemas in prompts, leading to long context lengths and increased prefilling latency. While user queries typically focus on recurrent table sets-offering an opportunity for KV cache sharing across queries-current inference engines, such as SGLang and vLLM, generate redundant prefix cache copies when processing user queries with varying table orders. To address this inefficiency, we propose precomputing table representations as KV caches offline and querying the required ones online. A key aspect of our approach is the computation of table caches while preserving primary foreign key relationships between tables. Additionally, we construct a Table Trie structure to facilitate efficient KV cache lookups during inference. To enhance cache performance, we introduce a cache management system with a query reranking strategy to improve cache hit rates and a computation loading pipeline for parallelizing model inference and cache loading. Experimental results show that our proposed TableCache achieves up to a 3.62x speedup in Time to First Token (TTFT) with negligible performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。