用智能体规划工具路径,让图像编辑更省钱省力。
CoSTA$\ast$: Cost-Sensitive Toolpath Agent for Multi-turn Image Editing
- 用大模型生成任务树,快速缩小搜索范围。
- 结合成本与质量评估,找到最优工具组合。
- 自动切换模态,适应不同编辑需求。
文本到图像模型如 Stable Diffusion 和 DALL-E 3 在多轮图像编辑任务中仍表现不佳。本文将此类任务分解为由不同成本的 AI 工具组成的智能体工作流(路径),通过一系列子任务完成。传统搜索算法需大量探索才能找到有效路径。虽然大语言模型(LLMs)具备子任务规划经验,但难以准确评估工具的能力与成本。为此,提出三阶段方法 CoSTA*:首先利用 LLM 构建子任务树以剪枝工具图,再在简化后的子图上运行 A* 搜索,寻找最优工具路径。为平衡总成本与质量,CoSTA* 在每一步融合工具的成本与质量指标引导搜索。每个子任务输出由视觉-语言模型(VLM)评估,若失败则更新对应工具在该任务上的成本与质量,使 A* 能快速回溯并探索新路径。此外,CoSTA* 可在子任务间自动切换模态以优化成本-质量权衡。我们构建了一个新的复杂多轮图像编辑基准,实验表明,CoSTA* 在成本与质量上均优于现有最先进模型或智能体,并可根据用户偏好实现灵活的权衡。
原文摘要 · Abstract (English)
Text-to-image models like stable diffusion and DALLE-3 still struggle with multi-turn image editing. We decompose such a task as an agentic workflow (path) of tool use that addresses a sequence of subtasks by AI tools of varying costs. Conventional search algorithms require expensive exploration to find tool paths. While large language models (LLMs) possess prior knowledge of subtask planning, they may lack accurate estimations of capabilities and costs of tools to determine which to apply in each subtask. Can we combine the strengths of both LLMs and graph search to find cost-efficient tool paths? We propose a three-stage approach "CoSTA*" that leverages LLMs to create a subtask tree, which helps prune a graph of AI tools for the given task, and then conducts A* search on the small subgraph to find a tool path. To better balance the total cost and quality, CoSTA* combines both metrics of each tool on every subtask to guide the A* search. Each subtask's output is then evaluated by a vision-language model (VLM), where a failure will trigger an update of the tool's cost and quality on the subtask. Hence, the A* search can recover from failures quickly to explore other paths. Moreover, CoSTA* can automatically switch between modalities across subtasks for a better cost-quality trade-off. We build a novel benchmark of challenging multi-turn image editing, on which CoSTA* outperforms state-of-the-art image-editing models or agents in terms of both cost and quality, and performs versatile trade-offs upon user preference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。