统一多模态表征，让文本、音频、图像等数据共用一套编码。
PixelBytes: Catching Unified Representation for Multimodal Generation
- 将文本、音频、动作、像素图整合为统一序列表示。
- 自回归模型在多模态生成上优于预测模型，效果更佳。
- 适用于游戏开发、智能控制等需要多模态生成的场景。
本文提出PixelBytes，一种统一的多模态表征学习方法。受Image Transformers、PixelCNN和Mamba-Bytes等序列模型启发，我们将文本、音频、动作状态及像素化图像（精灵图）整合为连贯的表征。在PixelBytes Pokemon数据集和Optimal-Control数据集上，测试了RNN、状态空间模型（SSMs）和基于注意力的模型，重点研究双向处理与PxBy嵌入技术。评估涵盖数据压缩策略与自回归学习，特别对比了长短期记忆网络（LSTM）在预测与自回归模式下的表现。结果表明，自回归模型在此任务中优于预测模型。此外，扩散模型可应用于控制问题并实现并行生成。该工作旨在推动多模态基础模型的发展。项目代码、模型与数据集均已公开。
原文摘要 · Abstract (English)
This report presents PixelBytes, an approach for unified multimodal representation learning. Drawing inspiration from sequence models like Image Transformers, PixelCNN, and Mamba-Bytes, we explore integrating text, audio, action-state, and pixelated images (sprites) into a cohesive representation. We conducted experiments on a PixelBytes Pokemon dataset and an Optimal-Control dataset. Our investigation covered various model architectures, including Recurrent Neural Networks (RNNs), State Space Models (SSMs), and Attention-based models, with a focus on bidirectional processing and our PxBy embedding technique. We evaluated models based on data reduction strategies and autoregressive learning, specifically examining Long Short-Term Memory (LSTM) networks in predictive and autoregressive modes. Our results indicate that autoregressive models perform better than predictive models in this context. Additionally, we found that diffusion models can be applied to control problems and parallelized generation. PixelBytes aims to contribute to the development of foundation models for multimodal data processing and generation. The project's code, models, and datasets are available online.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。