让小模型在资源有限时,提前猜下一步怎么走,加快推理速度。
B-PASTE: Beam-Aware Pattern-Guided Speculative Execution for Resource-Constrained LLM Agents
- 基于未来执行路径的猜想,用束搜索机制预判多条分支。
- 在边缘设备上实测提速1.4倍,且不抢关键任务资源。
- 适合部署在算力紧张的终端设备,如手机、嵌入式系统。
LLM智能体在推理与行动交替进行时,因串行依赖导致端到端延迟高,模型空等工具执行。现有方法PASTE仅预测单一工具调用,无法应对复杂分支。本文提出B-PASTE,通过维护有限束宽的未来执行子图,按预期关键路径缩减量而非执行概率排序,仅将高价值分支前缀调度至临时空闲资源。该方法显式建模并发干扰、下游解锁价值与状态安全性,可在早期完成时优先串行快速推进,低竞争时安全并行。特别适用于边缘部署场景,确保推测任务不占用延迟敏感的关键资源。内部测试显示,在Thor类边缘环境可实现最高1.4倍的端到端加速,证明其在严苛资源约束下仍有效。
原文摘要 · Abstract (English)
LLM agents execute in an interleaved reasoning-and-action loop, where future tool calls cannot be launched until the current reasoning step completes. This serial dependency inflates end-to-end latency and leaves the model idle while waiting for tool execution. Prior work, Pattern-Aware Speculative Tool Execution (PASTE), mitigates this bottleneck by speculating likely future tool invocations from mined control-flow and data-flow regularities. However, PASTE is tool-centric and speculates only individual invocations rather than bounded future branches. We propose B-PASTE, a beam-aware extension that lifts speculation from single tools to local branch hypotheses under strict resource constraints. B-PASTE maintains a bounded beam of future execution subgraphs, ranks them by expected critical-path reduction rather than raw execution probability, and schedules only high-value branch prefixes on transient slack resources. It explicitly models co-run interference, downstream unlock value, and state-safety constraints, enabling the system to prioritize serial fast-path execution when early completion unlocks valuable future work, while still exploiting safe parallelism under low contention. This design is especially important for edge-side deployments, where speculative work must not steal scarce resources from latency-critical authoritative execution. Preliminary internal testing on Thor-class edge environments shows up to 1.4X end-to-end speedup, suggesting that branch-aware speculative execution remains effective even under tight resource budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。