无需训练，让大模型直接看图听声并生成内容。
LLMs can see and hear without any training
- 用多轮推理迭代优化输出，不需额外训练
- 零样本图像/视频/音频生成均达新最好效果
- 适合想快速部署多模态能力的研究者
我们提出MILS：多模态迭代大模型求解器，一种无需训练的简单方法，可为任意大语言模型赋予多模态能力。利用其多步推理能力，MILS通过提示生成候选输出，逐轮评分并反馈，最终完成任务。该方法可实现通常需要专用模型训练的应用。特别地，我们在零样本图像、视频和音频字幕生成上建立新基准。MILS还可用于媒体生成，发现优化文本到图像生成的提示，甚至实现风格迁移提示编辑。作为无梯度优化方法，它能将多模态嵌入反向还原为文本，支持跨模态算术等应用。
原文摘要 · Abstract (English)
We present MILS: Multimodal Iterative LLM Solver, a surprisingly simple, training-free approach, to imbue multimodal capabilities into your favorite LLM. Leveraging their innate ability to perform multi-step reasoning, MILS prompts the LLM to generate candidate outputs, each of which are scored and fed back iteratively, eventually generating a solution to the task. This enables various applications that typically require training specialized models on task-specific data. In particular, we establish a new state-of-the-art on emergent zero-shot image, video and audio captioning. MILS seamlessly applies to media generation as well, discovering prompt rewrites to improve text-to-image generation, and even edit prompts for style transfer! Finally, being a gradient-free optimization approach, MILS can invert multimodal embeddings into text, enabling applications like cross-modal arithmetic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。