TMO实现设备云端协同推理，高效处理多模态多轮对话任务。
Device-Cloud Collaborative LLM Inference with Multi-Modal, Multi-Task, Multi-Turn Conversations
- 采用轻量设备模型与大型云模型协同，支持多模态输入和多轮对话。
- 通过资源受限强化学习优化任务调度，降低延迟37%、成本42%，提升响应质量。
- 适用于移动端部署场景，适合需要低延迟与低成本的智能交互应用。
相较于传统机器学习模型，大语言模型（LLMs）可通过多模态数据源和多轮对话实现多任务求解。然而，其大规模参数带来的计算、内存和能耗挑战使得部署困难：设备端受限于资源，云侧则难以保证实时性并产生通信与使用成本。本文提出TMO系统，支持设备-云协同的三重异构（多模态、多任务、多轮对话）推理。TMO包含一个轻量级本地LLM用于快速处理简单任务，以及一个大规模云端LLM处理复杂多模态任务。我们设计了资源约束强化学习（RCRL）策略，动态决策每轮任务的执行位置（设备或云端）及所需多模态数据源，以最大化长期收益（响应质量、延迟与成本），同时满足资源限制。我们还构建了M4A1数据集，覆盖多模态、多任务、多轮对话与多种模型配置，用于评估卸载策略。实验表明，TMO相比多种基线方法，在延迟、成本与响应质量上均有显著提升。代码与数据集已开源：https://github.com/liangqiyuan/TMO。
原文摘要 · Abstract (English)
Compared to traditional machine learning models, recent large language models (LLMs) can exhibit multi-task-solving capabilities through multi-modal data sources and multi-turn conversations. These unique characteristics of LLMs, together with their large model size, make their deployment more challenging. Specifically, (i) deploying LLMs on devices faces computational, memory, and energy resource issues, while (ii) deploying them in the cloud cannot guarantee real-time service and incurs communication/usage costs. In this paper, we design TMO, a device-cloud LLM inference system with Three-M Offloading: Multi-modal, Multi-task, and Multi-turn. TMO incorporates (i) a lightweight on-device LLM that can process simple tasks at high speed and (ii) a large-scale cloud LLM that can handle multi-modal data sources. We develop a resource-constrained reinforcement learning (RCRL) strategy for TMO that optimizes the inference location (i.e., device vs. cloud) and multi-modal data sources to use for each task in multi-turn conversations, aiming to maximize the long-term reward (response quality, latency, and usage cost) while adhering to resource constraints. We also contribute M4A1, a new dataset we curated across multiple modalities, tasks, conversation turns, and LLM configurations, enabling evaluation of offloading decisions. We demonstrate the effectiveness of TMO compared to several exploration-decision and LLM-as-Router baselines, showing significant improvements in latency, cost, and response quality. Our code and dataset are available at https://github.com/liangqiyuan/TMO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。