TinyServe通过查询感知缓存选择，实现小模型高效推理。
TinyServe: Query-Aware Cache Selection for Efficient LLM Serving
- 根据查询与缓存块的相关性，动态选择加载的键值缓存页。
- 实测推理速度提升3.4倍，内存减少超2倍，精度损失极小。
- 适合资源受限设备上的大模型推理研究与部署。
由于自回归解码中键值（KV）缓存访问带来高内存和延迟开销，大语言模型（LLMs）的高效服务仍具挑战。我们提出轻量级可扩展的服务系统TinyServe，支持部署小型语言模型（如TinyLLaMA、GPT2-345M），具备结构化KV稀疏、插件式标记选择和硬件高效的注意力核。与以往仿真框架不同，TinyServe可实时执行解码，并配置稀疏策略与细粒度监控。为降低解码成本，引入查询感知页选择机制，利用边界框元数据估算查询与KV缓存块间的注意力相关性，实现低开销的精准缓存加载，无需修改模型。融合CUDA核将页面评分、稀疏内存访问与掩码注意力合并为单次遍历。实验表明，TinyServe在保持精度几乎不变的前提下，实现最高3.4倍加速和超过2倍内存节省。对缓存复用率、页命中率及多GPU扩展性的分析进一步验证其在资源受限硬件上的实用性。
原文摘要 · Abstract (English)
Serving large language models (LLMs) efficiently remains challenging due to the high memory and latency overhead of key-value (KV) cache access during autoregressive decoding. We present \textbf{TinyServe}, a lightweight and extensible serving system for deploying tiny LLMs (e.g., TinyLLaMA, GPT2-345M) with support for structured KV sparsity, plugin-based token selection, and hardware-efficient attention kernels. Unlike prior simulation frameworks, TinyServe executes real-time decoding with configurable sparsity strategies and fine-grained instrumentation. To reduce decoding cost, we introduce a \textit{query-aware page selection} mechanism that leverages bounding-box metadata to estimate attention relevance between the query and KV cache blocks. This enables selective KV loading with minimal overhead and no model modifications. Our fused CUDA kernel integrates page scoring, sparse memory access, and masked attention in a single pass. Experiments show that TinyServe achieves up to \textbf{3.4x} speedup and over \textbf{2x} memory savings with negligible accuracy drop. Additional analysis of cache reuse, page hit rate, and multi-GPU scaling confirms its practicality as an efficient system-level design for LLM training and inference research on resource-constrained hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。