用专家路由机制自动评估图像编辑指令,结果更接近人类判断。
Routing to the Right Expertise: A Trustworthy Judge for Instruction-based Image Editing
- 通过动态路由选择擅长的模型专家进行评价
- 在多个数据集上与人工评价相关性达0.9以上
- 可解释性强,适合需要可信评估的AI研究者
基于指令的图像编辑(IIE)模型因多模态大模型和扩散模型的发展取得了显著进步,能够理解并推理复杂编辑指令。然而,准确评估其输出变得愈发关键且困难。现有评估方法常与人类判断不一致,且缺乏可解释性。为此,我们提出JURE:通过路由专家机制进行判断。每个专家是预先选定、具备特定原子能力的模型,路由器将给定指令及输出动态分配至合适专家,聚合反馈生成最终评价。JURE具有双重可信性:一是可通过查看被路由专家及其反馈提供可解释的判断依据;二是实验表明其与人类判断高度一致,相关系数超0.9,在多个基准上达到新标准。此外,其模块化设计可灵活替换或扩展专家,适应未来IIE发展,持续保持高评价质量。数据与代码已公开于https://github.com/Cyyyyyrus/JURE.git。
原文摘要 · Abstract (English)
Instruction-based Image Editing (IIE) models have made significantly improvement due to the progress of multimodal large language models (MLLMs) and diffusion models, which can understand and reason about complex editing instructions. In addition to advancing current IIE models, accurately evaluating their output has become increasingly critical and challenging. Current IIE evaluation methods and their evaluation procedures often fall short of aligning with human judgment and often lack explainability. To address these limitations, we propose JUdgement through Routing of Expertise (JURE). Each expert in JURE is a pre-selected model assumed to be equipped with an atomic expertise that can provide useful feedback to judge output, and the router dynamically routes the evaluation task of a given instruction and its output to appropriate experts, aggregating their feedback into a final judge. JURE is trustworthy in two aspects. First, it can effortlessly provide explanations about its judge by examining the routed experts and their feedback. Second, experimental results demonstrate that JURE is reliable by achieving superior alignment with human judgments, setting a new standard for automated IIE evaluation. Moreover, JURE's flexible design is future-proof - modular experts can be seamlessly replaced or expanded to accommodate advancements in IIE, maintaining consistently high evaluation quality. Our evaluation data and results are available at https://github.com/Cyyyyyrus/JURE.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。