arXiv:2509.00195cs.LG2025-09中稿 · ASPLOS 2026被引 7

让边缘设备上的小模型也能快速推理,媲美云端大模型。

FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning

  • 通过动态调度与内存优化,提升边缘端推理效率。
  • 实测延迟降低38%~68%,吞吐量提升2.2倍。
  • 适合希望在手机/本地设备部署智能代理的开发者。

近期推理型大语言模型(LLM)的发展推动了智能体(agentic AI)系统的兴起。将LLM智能体部署在靠近用户的边缘设备上,有助于保护数据隐私、支持离线使用并实现对本地上下文的快速响应。然而,边缘设备严格的内存限制导致只能部署小型模型,其推理能力远弱于云端大模型,制约了边缘智能体的实际应用。测试时扩展(Test-Time Scaling, TTS)通过在推理阶段分配更多计算资源来增强小模型的推理能力,但现有TTS方法在资源受限设备上引入显著硬件性能开销,难以实用。为此,我们提出FastTTS,一个面向内存受限边缘推理的高效服务系统。通过对多种TTS方法共性模式分析并识别性能瓶颈,提出三项新技术:i)推测性束扩展,缓解不规则推理路径引发的系统延迟;ii)非对称多模型内存分配,动态平衡生成与验证阶段的内存使用;iii)动态前缀感知调度,优化推理执行以最大化跨搜索路径的键值缓存复用。FastTTS作为vLLM的即插即用库,使单个消费级GPU上的边缘模型达到云端模型的准确率和云测延迟。全面评估表明,相较于vLLM基线,FastTTS平均提升2.2倍吞吐量,延迟降低38%–68%,突破了内存受限设备上低延迟TTS的边界,展现了智能体AI普惠化的潜力。

原文摘要 · Abstract (English)

Recent advances in reasoning Large Language Models (LLMs) are driving the emergence of agentic AI systems. Edge deployment of LLM agents near end users is increasingly necessary to protect data privacy, enable offline use, and provide responsive interaction with local context. However, strict memory constraints on edge devices limit deployment to smaller LLMs, whose reasoning capabilities are much weaker than those of large cloud models, hindering practical deployment of edge agentic AI. Test-Time Scaling (TTS) offers a promising solution by allocating more compute during inference to enhance the reasoning capability of edge LLMs. However, current TTS methods introduce heavy hardware performance overhead on resource-constrained devices, making them impractical for real applications. To address this challenge, we present FastTTS, a serving system that enables fast and efficient TTS for memory-constrained LLM reasoning. After analyzing common patterns across various TTS methods and identifying their performance bottlenecks, we introduce three novel techniques: i) Speculative Beam Extension, which mitigates system stragglers caused by irregular reasoning paths, ii) Asymmetric Multi-Model Memory Allocation, which dynamically balances memory usage between token generation and reasoning-step verification, and iii) Dynamic Prefix-Aware Scheduling, which optimizes reasoning execution to maximize KV-cache reuse across search paths. FastTTS offers a plug-and-play third-party library on top of vLLM, enabling edge LLMs on a single consumer GPU to match cloud-model accuracy and cloud-measured latency. Comprehensive evaluation shows that FastTTS achieves an average 2.2x higher goodput and reduces latency by 38%--68% compared to the vLLM baseline; it pushes the boundaries of low-latency TTS on memory-constrained edge devices and highlights the potential for democratizing agentic AI.

边缘计算推理加速TTSLLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。