让大模型生成代码时同步执行，大幅降低推理延迟。
Executing as You Generate: Hiding Execution Latency in LLM Code Interpreters
- 生成代码同时分段执行，利用前后文顺序性实现流水线并行。
- 错误无损情况下端到端延迟降低37.3%，执行重叠率达99.8%。
- 适合需要低延迟代码推理的部署场景，如实时交互系统。
当前大模型系统普遍配备代码解释器，但采用串行流程：先生成完整代码，再执行。这导致生成与执行阶段各自空闲，造成不必要的端到端延迟。我们观察到，大模型生成代码时从左到右逐令牌输出且不回溯，因此可在后续令牌生成的同时提前执行已生成部分。我们提出一种三阶段并行执行范式（生成-检测-执行），推导出闭式延迟边界，刻画其加速潜力与运行区间。进而提出EAGER实现，包含基于抽象语法树的分块、带门控的动态批处理与早期错误中断机制。在四个基准、七种大模型及三种执行环境上评估，该机制几乎完全隐藏执行延迟，错误无损运行下非重叠执行时间减少99.8%，端到端延迟最高降低37.3%。
原文摘要 · Abstract (English)
Current LLM systems are increasingly equipped with a code interpreter that executes generated code to obtain results. This works serially: the model first generates the complete code, then an interpreter executes it. This sequential workflow leaves the executor idle during generation and the generator idle during execution, resulting in unnecessary end-to-end latency. Our key observation is that an LLM, unlike a human developer, emits code tokens left to right and does not backtrack over what it has already written. This makes it possible to start executing a piece of code while later tokens are still being generated. We formalize this parallel execution paradigm, modeling it as a three-stage pipeline of generation, detection, and execution, and derive closed-form latency bounds that characterize its speedup potential and operating regimes. We then present EAGER, a concrete implementation featuring AST-based chunking, dynamic batching with gated execution, and early error interruption. We evaluate EAGER across four benchmarks, seven LLMs, and three execution environments. The overlap mechanism hides almost all execution behind generation, reducing the non-overlapped portion of execution time by up to 99.8% and cutting end-to-end latency by up to 37.3% on error-free runs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。