轻量级多模态模型，让手机实时完成图文理解与生成。
Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device
- 用深度可分离卷积融合视觉语言特征，高效驱动图像生成。
- 仅用数百万样本训练，生成与理解性能均超同类模型。
- 手机端3秒内完成512x512图像处理，适合无云依赖的移动端应用。
统一多模态模型可在单一架构中同时实现视觉内容的理解与生成。然而，现有模型仍需大量数据且计算量过大，难以部署于边缘设备。本文提出 Mobile-O，一个紧凑的视觉-语言-扩散模型，首次将统一多模态智能落地至移动设备。其核心模块 Mobile Conditioning Projector（MCP）采用深度可分离卷积与逐层对齐，高效融合视觉-语言特征并驱动扩散生成器，显著降低计算开销。模型仅在数百万样本上训练，并通过新型四元组格式（生成提示、图像、问题、答案）进行后训练，协同提升理解与生成能力。尽管体积小，Mobile-O 在 GenEval 上达到74%得分，优于 Show-O 和 JanusFlow 5%和11%，推理速度分别快6倍和11倍；在七个基准上平均理解性能领先15.3%和5.1%。单张512x512图像处理时间仅约3秒（iPhone），建立首个可在边缘设备上实时运行的统一多模态框架。代码、模型、数据集及移动应用已开源。
原文摘要 · Abstract (English)
Unified multimodal models can both understand and generate visual content within a single architecture. Existing models, however, remain data-hungry and too heavy for deployment on edge devices. We present Mobile-O, a compact vision-language-diffusion model that brings unified multimodal intelligence to a mobile device. Its core module, the Mobile Conditioning Projector (MCP), fuses vision-language features with a diffusion generator using depthwise-separable convolutions and layerwise alignment. This design enables efficient cross-modal conditioning with minimal computational cost. Trained on only a few million samples and post-trained in a novel quadruplet format (generation prompt, image, question, answer), Mobile-O jointly enhances both visual understanding and generation capabilities. Despite its efficiency, Mobile-O attains competitive or superior performance compared to other unified models, achieving 74% on GenEval and outperforming Show-O and JanusFlow by 5% and 11%, while running 6x and 11x faster, respectively. For visual understanding, Mobile-O surpasses them by 15.3% and 5.1% averaged across seven benchmarks. Running in only ~3s per 512x512 image on an iPhone, Mobile-O establishes the first practical framework for real-time unified multimodal understanding and generation on edge devices. We hope Mobile-O will ease future research in real-time unified multimodal intelligence running entirely on-device with no cloud dependency. Our code, models, datasets, and mobile application are publicly available at https://amshaker.github.io/Mobile-O/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。