一站式解析多模态大模型技术与调优方法
Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
- 系统梳理视觉、语言、音频等多模态融合的技术路径
- 涵盖跨模态预训练模型及指令微调的最新进展
- 适合想快速入门多模态AI的研究者与开发者
本教程探讨多模态预训练与大模型的最新进展，能够整合处理文本、图像、音频、视频等多种数据形式。参与者将了解多模态的基础概念、研究演进历程及关键挑战。内容包括最新的多模态数据集与预训练模型，涵盖视觉与语言之外的模态。还将深入讲解多模态大模型的结构设计与指令调优策略，以提升特定任务性能。通过动手实验，体验如视觉故事生成、视觉问答等实际应用。本教程旨在为研究人员、从业者及初学者提供掌握多模态人工智能的知识与技能。ACM Multimedia 2024是理想的举办场合，契合我们对多模态预训练大模型及其调优机制的理解目标。
原文摘要 · Abstract (English)
This tutorial explores recent advancements in multimodal pretrained and large models, capable of integrating and processing diverse data forms such as text, images, audio, and video. Participants will gain an understanding of the foundational concepts of multimodality, the evolution of multimodal research, and the key technical challenges addressed by these models. We will cover the latest multimodal datasets and pretrained models, including those beyond vision and language. Additionally, the tutorial will delve into the intricacies of multimodal large models and instruction tuning strategies to optimise performance for specific tasks. Hands-on laboratories will offer practical experience with state-of-the-art multimodal models, demonstrating real-world applications like visual storytelling and visual question answering. This tutorial aims to equip researchers, practitioners, and newcomers with the knowledge and skills to leverage multimodal AI. ACM Multimedia 2024 is the ideal venue for this tutorial, aligning perfectly with our goal of understanding multimodal pretrained and large language models, and their tuning mechanisms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。