通过跳过冗余计算，显著降低多模态大模型推理开销。
Skipping Computations in Multimodal LLMs
- 提出跳过完整模块、前馈层或自注意力层的计算策略。
- 跳过一半模块或70%权重后，性能仍保持原模型97%。
- 小模型经优化训练，可达到大模型2-3倍规模的效果。
大型语言模型在文本和多模态领域表现卓越，但处理长序列多模态输入时计算成本高昂。本文研究多模态大模型推理中的计算冗余问题，提出跳过整个模块、前馈网络（FFN）或自注意力（SA）层的计算方法，并探索并行化某些层的可行性。实验表明：（1）在视觉问答（VQA）等任务中，可显著减少推理阶段的计算量；（2）训练时跳过一半模块或删除70%权重，性能仍可恢复至原始性能的97%；（3）经过合理训练的小模型，性能可媲美2至3倍大小的大型模型。我们在最新多模态模型如LLaVA-1.5上验证了相似现象。结果表明，多模态大模型中存在冗余计算，有潜力在不牺牲性能的前提下大幅降低推理开销。代码已公开于https://github.com/mshukor/ima-lmms。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable success in both textual and multimodal domains. However, this success often comes with substantial computational costs, particularly when handling lengthy sequences of multimodal inputs. This has sparked many efforts focusing on enhancing efficiency during training and inference. In this study, we investigate the computation redundancy in Multimodal Large Language Models (MLLMs) during inference. We propose different methods to skip computations, such as skipping entire blocks, FFN or self-attention (SA) layers. Additionally, we explore parallelizing certain layers, such as FFN and SA layers. Our findings validate that (1) significant amount of computations can be avoided at inference time, especially for tasks such as Visual Question Answering (VQA). (2) Skipping computations during training can recover 97% of the original performance, even when skipping half of the blocks or removing 70% of the weights. Alternatively, (3) properly training with smaller LLMs can yield comparable performance to LLMs 2 or 3 times larger. To conclude, we extend our investigation to recent MLLMs, such as LLaVA-1.5, showing similar observations. Our work show that there is redundant computations inside MLLMs and thus the potential for significantly improving inference costs without sacrificing performance. The code is available here: https://github.com/mshukor/ima-lmms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。