用下一个词预测统一多模态智能，打通视觉与语言建模
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
- 以下一个词预测为核心框架，统一多模态理解与生成任务
- 构建涵盖五方面的系统性分类体系，覆盖从分词到评估全链路
- 适合想快速入门多模态大模型的研究者和开发者
基于自然语言处理中的语言建模基础，下一个词预测（NTP）已发展为跨模态机器学习任务的通用训练目标，并取得显著成果。随着大语言模型（LLM）在文本模态中实现理解与生成的统一，近期研究发现，不同模态的任务也可有效纳入NTP框架，将多模态信息转换为可预测的token序列。本文提出一个全面的分类体系，从NTP视角统一多模态学习中的理解和生成任务。该分类涵盖五个核心方面：多模态分词、MMNTP模型架构、统一任务表示、数据集与评估方法，以及开放挑战。该体系旨在帮助研究人员推进多模态智能发展。相关论文与代码仓库已整理至GitHub：https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction。
原文摘要 · Abstract (English)
Building on the foundations of language modeling in natural language processing, Next Token Prediction (NTP) has evolved into a versatile training objective for machine learning tasks across various modalities, achieving considerable success. As Large Language Models (LLMs) have advanced to unify understanding and generation tasks within the textual modality, recent research has shown that tasks from different modalities can also be effectively encapsulated within the NTP framework, transforming the multimodal information into tokens and predict the next one given the context. This survey introduces a comprehensive taxonomy that unifies both understanding and generation within multimodal learning through the lens of NTP. The proposed taxonomy covers five key aspects: Multimodal tokenization, MMNTP model architectures, unified task representation, datasets \& evaluation, and open challenges. This new taxonomy aims to aid researchers in their exploration of multimodal intelligence. An associated GitHub repository collecting the latest papers and repos is available at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。