让多模态大模型自适应调整推理，省资源还保效果。
Learning to Inference Adaptively for Multimodal Large Language Models
- 训练模型动态调整推理过程，根据输入和延迟预算实时优化
- 在多个任务上实现不同精度与延迟的灵活权衡，满足实时需求
- 适配不同模型和输入内容，适合边缘设备部署
多模态大语言模型（MLLMs）在视觉推理方面表现优异，但计算开销大，限制了在资源受限场景的应用。尽管已有研究提升效率，但现有方法难以应对运行时资源波动（如设备上其他程序竞争资源）。为此，我们提出AdaLLaVA，一种自适应推理框架，通过学习在推理过程中动态重配置MLLM的操作，以响应输入数据和延迟预算。我们在包含问答、推理和幻觉检测的多个基准上进行广泛实验。结果表明，AdaLLaVA能有效遵守输入延迟预算，在运行时实现不同的准确率与延迟权衡。进一步验证了其对延迟和内容变化的适应性，可与令牌选择结合以提升效率，并具备跨MLLM的泛化能力。项目网页及代码已开源：https://zhuoyan-xu.github.io/ada-llava/
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in visual reasoning, yet come with substantial computational cost, limiting their deployment in resource-constrained settings. Despite recent effort on improving the efficiency of MLLMs, prior solutions fall short in responding to varying runtime conditions, in particular changing resource availability (e.g., contention due to the execution of other programs on the device). To bridge this gap, we introduce AdaLLaVA, an adaptive inference framework that learns to dynamically reconfigure operations in an MLLM during inference, accounting for the input data and a latency budget. We conduct extensive experiments across benchmarks involving question-answering, reasoning, and hallucination. Our results show that AdaLLaVA effectively adheres to input latency budget, achieving varying accuracy and latency tradeoffs at runtime. Further, we demonstrate that AdaLLaVA adapts to both input latency and content, can be integrated with token selection for enhanced efficiency, and generalizes across MLLMs. Our project webpage with code release is at https://zhuoyan-xu.github.io/ada-llava/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。