让大模型推理时实时观察内部状态更高效，延迟降低2到15倍。
Enabling Performant and Flexible Model-Internal Observability for LLM Inference
- 用异步机制将观测与推理主流程分离，避免性能损耗。
- 批量推理开销仅0.4%~6.8%，在线服务平均延迟增加6%。
- 适合需要调试、监控或优化大模型推理的开发者使用。
当前推理任务越来越依赖对模型内部状态的及时访问。我们提出DMI-Lib，一种高速深度模型观测工具，将内部可观测性作为系统核心原语，通过基于Ring^2（一种用于捕获和暂存张量的GPU-CPU内存抽象）构建的异步可观测性底座，以及策略控制的主机后端，将观测过程从推理热路径中解耦。DMI-Lib支持在丰富多样的内部信号和不同推理后端中灵活设置观测点，同时保持服务优化并严格遵守GPU内存预算。实验表明，DMI-Lib在离线批处理推理中引入0.4%–6.8%开销，在中等负载在线服务中平均增加6%延迟，相比具备类似可观测功能的现有基线，延迟开销降低2–15倍。代码已开源：https://github.com/ProjectDMX/DMI。
原文摘要 · Abstract (English)
Today's inference-time workloads increasingly depend on timely access to a model's internal states. We present DMI-Lib, a high-speed deep model inspector that treats internal observability as a first-class systems primitive, decoupling it from the inference hot path via an asynchronous observability substrate built from Ring^2, a GPU-CPU memory abstraction for capturing and staging tensors, and a policy-controlled host backend that exports them. DMI-Lib enables the placement of observation points across a rich space of internal signals and diverse inference backends while preserving serving optimizations and adhering to tight GPU memory budgets. Our experiments demonstrate that DMI-Lib incurs only 0.4%--6.8% overhead in offline batch inference and an average of 6% in moderate online serving, reducing latency overhead by 2x-15x compared to existing baselines with similar observability features. DMI-Lib is open-sourced at https://github.com/ProjectDMX/DMI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。