Dooly让大模型推理模拟更快更准,一次运行就能适配多种配置。
Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation

- 通过追踪输入维度来源,自动识别可复用的计算路径,避免重复测量。
- 在12个模型上减少56.4%的显卡耗时,TTFT误差小于5%,TPOT误差小于8%。
- 无需手动改代码,可直接接入现有模拟器,适合做推理优化的研究者和工程师。
选择最优的大语言模型推理配置需在硬件、服务引擎、注意力后端和模型架构间进行评估,因为没有一种方案在所有负载下表现最佳。基于性能分析的模拟器是标准工具,但它们将操作集固定于特定配置,每次需从头重新分析,导致探索成本极高。其根源在于缺乏对结构的理解:每个操作的输入维度要么由模型配置决定,要么由请求确定。许多模型配置参数(如头大小、层数)在不同模型中重复出现,使得相同操作在多个配置下运行;一次遍历请求相关维度即可覆盖所有情况。我们提出 Dooly,利用该结构实现配置无关、冗余感知的性能分析。Dooly 执行一次推理,通过污点传播标注每个输入维度的来源,并仅对未在延迟数据库中的操作进行采样;状态化操作(如注意力)通过复用服务引擎自身初始化代码隔离,无需手动注入。它基于数据库构建延迟回归模型,可作为现有模拟器的即插即用后端。在两个GPU平台、三种注意力后端和多种模型架构上,Dooly 实现了 TTFT 的 5% 以内 MAPE 和 TPOT 的 8% 以内误差,相比现有方法在12个模型上降低 56.4% 的分析显卡小时。项目已开源:https://github.com/dooly-project。
原文摘要 · Abstract (English)
Selecting the optimal LLM inference configuration requires evaluation across hardware, serving engines, attention backends, and model architectures, since no single choice performs best across all workloads. Profile-based simulators are the standard tool, yet they hardcode their operation set to a specific configuration and re-profile every operation from scratch, making exploration prohibitively expensive. This cost stems from a missing structural understanding: every input dimension of each operation is fixed by the model configuration or determined by the incoming request. Many model-configuration values (e.g., head size, layer count) recur across models, so the same operation runs in many configurations; a single sweep over the request-dependent dimensions can serve them all. We present Dooly, which exploits this structure to achieve configuration-agnostic, redundancy-aware profiling. Dooly performs a single inference pass, labels each input dimension with its origin via taint propagation, and selectively profiles only operations absent from its latency database; stateful operations such as attention are isolated by reusing the serving engine's own initialization code, eliminating manual instrumentation. It builds latency regression models based on the database, which becomes a drop-in backend for existing simulators. Across two GPU platforms, three attention backends, and diverse model architectures, Dooly achieves simulation accuracy within 5% MAPE for TTFT and 8% for TPOT while reducing profiling GPU-hours by 56.4% across 12 models compared to the existing profiling approach. We have open-sourced Dooly at https://github.com/dooly-project.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。