通过逐令牌分析,动态决定模型路由,提升多模态大模型推理效率。
Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference

- 基于令牌级置信度进行分步路由决策,避免盲目调用大模型。
- 路由速度提升10倍以上,端到端吞吐量提高75%以上。
- 自适应边缘-云协同架构,无需人工调参,适配不同网络环境。
多模态大语言模型虽性能卓越,但计算开销巨大,难以实现实时部署与成本控制。现有路由方法或仅依赖请求级粗粒度特征,或需额外语言模型推理来评估输出,未能利用生成过程中的令牌级不确定性信号。为此,我们提出 Pro-Router,一种具备令牌感知的渐进式模型路由方法,支持自适应边缘-云协作,实现高效的多模态大模型推理。Pro-Router采用两阶段渐进决策机制:首先,轻量级提示预评分模块在生成前快速筛选简单请求,引导其使用小模型;其次,令牌感知验证器读取小模型生成每个令牌的概率分布,评估其输出置信度,按请求决定是否将答案直接返回或升级至云端高精度模型。此外,设计了自适应边缘-云服务管道,根据各设备实际服务速率动态分配任务,使边缘与云端始终高效运行,无需手动调参且不受网络延迟影响。在多个多模态基准数据集和模型上的实验表明,Pro-Router在路由准确率上达到最高,路由速度提升超过10倍,服务管道端到端吞吐量较现有方案提升75%以上。代码已开源:https://github.com/xinyuangui2/pro-router。
原文摘要 · Abstract (English)
The remarkable performance of multimodal large language models (MLLMs) comes at the cost of substantial computational overhead, posing significant challenges to real-time deployment and cost effectiveness. Existing model routing approaches either decide from coarse request-level features alone or spend one or several extra language model passes to inspect the generated response, leaving the token-level uncertainty signals that emerge during generation unused. To address these limitations, we propose Pro-Router, a token-aware progressive model routing method with adaptive edge-cloud collaboration for efficient multimodal LLM inference. Pro-Router employs a two-stage progressive decision mechanism. First, a lightweight prompt pre-scorer module performs rapid pre-screening before token generation begins, guiding apparently simple requests to small models. Second, a token-aware verifier reads the sampling probability distribution of each token the small model generates, estimating the model's confidence in its own output to determine, per request, whether the answer ships or escalates to the cloud-based high-precision model. Furthermore, we design an adaptive edge-cloud serving pipeline that sizes every dispatch to each device's measured service rate, so both the edge and the cloud tiers stay fully utilized without manual parameter tuning and are not impacted by the network latency. Extensive experiments on multiple multimodal benchmark datasets and models demonstrate the effectiveness of Pro-Router. Compared to other methods, it achieves the highest routing accuracy and improves routing speed by more than 10x. Its serving pipeline also reaches more than 75% higher end-to-end throughput than the existing model routing pipeline. Our code is available at https://github.com/xinyuangui2/pro-router.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。