让机制可解释性在生产环境低延迟运行,不拖慢大模型推理。
xMIx: High-Performance Serving-Time Platform for Mechanistic Interpretability Apps

- 在模型运行时预设位置插入可解释函数,动态触发不干扰主流程。
- 实测平均延迟仅增加1.3%,尾部延迟增加1.2%,性能接近原生推理。
- 支持多个可解释任务共存,无需额外实例或改造执行路径。
机制可解释性(MI)已成为分析和干预推理计算的强大方法,应用于越狱检测、真实性评估和幻觉识别等场景。然而,现有MI框架在生产推理服务中部署困难,因其带来过高运行时开销。根本问题在于MI函数与服务模型难以兼容:导致部署碎片化,常需清空请求并重建服务状态,并与连续批处理、CUDA图执行等关键性能优化冲突。我们提出xMIx,一个面向生产推理环境的原生可解释性部署框架。xMIx可在模型运行时预设位置插入MI函数,对层内激活和残差流进行介入。支持根据前序层输出条件性调用MI函数。多个MI应用可部署于同一模型实例中。xMIx将所有MI逻辑编译至服务路径,仅在必要时动态激活,性能损耗极小,且无需独立模型实例或另建执行栈。我们将其集成至vLLM系统,评估涵盖三个主流模型与七种不同MI应用。结果表明,xMIx实现与原生vLLM相近性能,平均分词延迟(ITL)仅提升1.3%,尾部99%分词延迟(P99 ITL)提升1.2%,平均首令牌时间(TTFT)增加2.6%,平均总令牌吞吐量(TTT)下降1.6%。
原文摘要 · Abstract (English)
Mechanistic interpretability (MI) has emerged as a powerful approach for analyzing and intervening in inference computations, with a growing number of applications such as jailbreak attempt detection, truthfulness evaluation, and hallucination detection. Unfortunately, MI deployment in production model-serving systems is currently not practical, as most existing MI frameworks introduce prohibitively high runtime overheads. The fundamental problem is that MI functions do not compose cleanly with served models: they fragment deployment, often force draining requests and rebuilding serving state, and conflict with critical performance optimizations such as continuous batching and CUDA-graph execution, essential for production deployments. We present xMIx, a serving-native framework for deploying MI applications in production inference serving environments. xMIx enables attaching MI functions to a predefined set of locations in the model runtime, interposing on activations within the layers and residual streams. xMIx supports conditional invocation of MI functions depending on the outputs in preceding model layers. Multiple MI applications can be deployed in a single model instance. xMIx compiles them all into the serving path but activates them dynamically at runtime only when necessary, with negligible performance cost, and without requiring a separate model instance or alternative execution stack. We integrate xMIx with the vLLM serving system and evaluate it across three major models and seven diverse MI applications. xMIx achieves performance comparable to native vLLM execution, incurring a slowdown of 1.3% mean inter-token latency (ITL), 1.2% for tail P99 ITL, 2.6% for mean time to first token (TTFT), and 1.6% for mean total token throughput (TTT).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。