轻量适配器让视频大模型轻松支持音频、3D等新模态。
PAVE: Patching and Adapting Video Large Language Models
- 通过添加微型'补丁'适配器，不改动原模型结构与权重。
- 在音视频问答、3D推理等任务上显著提升性能，额外开销仅0.1% FLOPs。
- 可跨模型通用，支持多任务学习，适合快速部署新模态应用。
预训练的视频大语言模型（Video LLMs）展现出强大的推理能力，但将其适配到包含新模态或数据类型（如音频或3D信息）的新任务仍具挑战。本文提出PAVE框架，用于将预训练的Video LLMs适配至包含侧通道信号（如音频、3D线索或多视角视频）的下游任务。PAVE引入轻量级适配器，称为“patches”，仅增加少量参数和计算量，无需修改基础模型架构或预训练权重。该方法有效使基座模型支持多种下游任务，包括音视频问答、3D推理、多视角视频识别和高帧率视频理解。在各项任务中，PAVE显著提升基座模型性能，超越现有专用模型，同时仅带来约0.1%的额外FLOPs和参数开销。此外，PAVE支持多任务学习，并在不同Video LLMs间具有良好泛化性。代码已公开于https://github.com/dragonlzm/PAVE。
原文摘要 · Abstract (English)
Pre-trained video large language models (Video LLMs) exhibit remarkable reasoning capabilities, yet adapting these models to new tasks involving additional modalities or data types (e.g., audio or 3D information) remains challenging. In this paper, we present PAVE, a flexible framework for adapting pre-trained Video LLMs to downstream tasks with side-channel signals, such as audio, 3D cues, or multi-view videos. PAVE introduces lightweight adapters, referred to as "patches," which add a small number of parameters and operations to a base model without modifying its architecture or pre-trained weights. In doing so, PAVE can effectively adapt the pre-trained base model to support diverse downstream tasks, including audio-visual question answering, 3D reasoning, multi-view video recognition, and high frame rate video understanding. Across these tasks, PAVE significantly enhances the performance of the base model, surpassing state-of-the-art task-specific models while incurring a minor cost of ~0.1% additional FLOPs and parameters. Further, PAVE supports multi-task learning and generalizes well across different Video LLMs. Our code is available at https://github.com/dragonlzm/PAVE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。