用自回归模型生成多视角一致图像，支持任意指令输入。
Auto-Regressively Generating Multi-View Consistent Images
- 采用自回归机制逐步生成多视角图像，利用已有视图提供参考信息。
- 在多种条件下（文本、相机位姿等）实现高质量多视角一致性生成。
- 提出视图打乱数据增强，有效缓解小样本过拟合问题，适合3D内容创作人群。
从人类指令生成多视角图像对3D内容创作至关重要。主要挑战在于保持多视角间的一致性，并在不同条件下有效合成形状与纹理。本文提出多视角自回归（MV-AR）方法，通过自回归模型从任意提示中渐进式生成一致的多视角图像。首先，自回归模型的下一个词预测能力显著提升其在渐进式多视角合成中的效果；在生成远距离视角时，可利用所有先前视角提取有效参考信息。其次，设计统一架构与训练策略，支持文本、相机位姿、图像和形状等多种提示条件。通过条件注入模块和渐进式训练策略，实现多模态条件的联合建模。最后，为缓解高质量数据有限导致的过拟合问题，提出“视图打乱”数据增强技术，使训练数据量显著扩大数个数量级。实验表明，MV-AR在各类条件下均能稳定生成一致的多视角图像，性能媲美领先的基于扩散模型的多视角图像生成方法。代码与模型已开源：https://github.com/MILab-PKU/MVAR。
原文摘要 · Abstract (English)
Generating multi-view images from human instructions is crucial for 3D content creation. The primary challenges involve maintaining consistency across multiple views and effectively synthesizing shapes and textures under diverse conditions. In this paper, we propose the Multi-View Auto-Regressive (\textbf{MV-AR}) method, which leverages an auto-regressive model to progressively generate consistent multi-view images from arbitrary prompts. Firstly, the next-token-prediction capability of the AR model significantly enhances its effectiveness in facilitating progressive multi-view synthesis. When generating widely-separated views, MV-AR can utilize all its preceding views to extract effective reference information. Subsequently, we propose a unified model that accommodates various prompts via architecture designing and training strategies. To address multiple conditions, we introduce condition injection modules for text, camera pose, image, and shape. To manage multi-modal conditions simultaneously, a progressive training strategy is employed. This strategy initially adopts the text-to-multi-view (t2mv) model as a baseline to enhance the development of a comprehensive X-to-multi-view (X2mv) model through the randomly dropping and combining conditions. Finally, to alleviate the overfitting problem caused by limited high-quality data, we propose the ``Shuffle View" data augmentation technique, thus significantly expanding the training data by several magnitudes. Experiments demonstrate the performance and versatility of our MV-AR, which consistently generates consistent multi-view images across a range of conditions and performs on par with leading diffusion-based multi-view image generation models. The code and models are released at https://github.com/MILab-PKU/MVAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。