让大模型像用工具一样灵活生成图文，突破单一模型局限
LLM-I: LLMs are Naturally Interleaved Multimodal Creators
- 将图文生成视为工具调用问题，动态组合搜索、生成、编辑等专用工具
- 在4个基准上超越现有方法，性能提升显著，且支持测试时扩展策略
- 适合需要精准事实或程序化控制的图文创作场景
我们提出LLM-Interleaved（LLM-I），一个灵活动态的框架，将交错的图文生成重新定义为工具使用问题。该框架旨在克服当前统一模型存在的“单一工具”瓶颈，这些模型仅限于合成图像，难以处理需要事实依据或程序精确性的任务。LLM-I使中心大模型或多模态大模型代理能够智能调度多样化的专业视觉工具，包括在线图像搜索、基于扩散的生成、代码执行和图像编辑。代理通过强化学习框架训练，采用结合规则逻辑与大模型及多模态大模型评估者判断的混合奖励系统，实现工具的高效选择与应用。在包含四种不同模型主干的新数据集上训练后，LLM-I在四个基准测试中均达到顶尖水平，性能远超现有方法。我们还引入一种新颖的测试时扩展策略，进一步提升表现。项目页面：https://github.com/ByteDance-BandAI/LLM-I。
原文摘要 · Abstract (English)
We propose LLM-Interleaved (LLM-I), a flexible and dynamic framework that reframes interleaved image-text generation as a tool-use problem. LLM-I is designed to overcome the "one-tool" bottleneck of current unified models, which are limited to synthetic imagery and struggle with tasks requiring factual grounding or programmatic precision. Our framework empowers a central LLM or MLLM agent to intelligently orchestrate a diverse toolkit of specialized visual tools, including online image search, diffusion-based generation, code execution, and image editing. The agent is trained to select and apply these tools proficiently via a Reinforcement Learning (RL) framework that features a hybrid reward system combining rule-based logic with judgments from LLM and MLLM evaluators. Trained on a diverse new dataset using four different model backbones, LLM-I demonstrates state-of-the-art performance, outperforming existing methods by a large margin across four benchmarks. We also introduce a novel test-time scaling strategy that provides further performance gains. Project Page: https://github.com/ByteDance-BandAI/LLM-I.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。