arXiv:2605.29461cs.CV2026-05中稿 · ICML

让语言条件动态指导分割,提升图文匹配准确率

FlowSeg: Dynamic Semantic Guidance for LLM-Conditioned Segmentation

论文配图:FlowSeg: Dynamic Semantic Guidance for LLM-Conditioned Segmentation
图 1 · 摘自论文原文
  • 引入双向语义流,语言与视觉信息实时交互优化
  • 在多个任务上实现最佳对齐效果,显著降低语义错配
  • 适合需要精准语义理解的图像分割场景

LLM 条件分割近年通过将大语言模型与迭代掩码生成框架结合而快速发展。然而我们发现当前“提出-选择”流程存在持续性失败模式:尽管生成了高质量掩码候选,最终预测仍可能不匹配给定语言条件。此问题源于语言语义通常作为静态提示或事后匹配信号,未参与迭代生成过程。系统分析表明,多数错误源于语义错位而非掩码质量差。为此,我们提出 FlowSeg,通过在中间解码状态与 LLM 推导的条件嵌入间建立双向语义流,实现动态语义引导。语言条件在每一步主动指导掩码精炼,同时条件嵌入由新出现的视觉证据逐步更新。该设计生成语义扎实的掩码表示与视觉对齐的语言条件,提升匹配可靠性。进一步引入轻量级边界感知精修模块,仅增强不确定区域而不扰动置信内部。在参考表达分割和推理分割任务上的大量实验表明,FlowSeg 持续改善语言-掩码对齐,达到当前最优性能。

原文摘要 · Abstract (English)

LLM-conditioned segmentation has recently advanced rapidly by coupling large language models with iterative mask generation frameworks. However, we identify a persistent failure mode in current propose-then-select pipelines. Although high-quality mask candidates are often generated, the final prediction may fail to match the given linguistic condition. This failure arises because language semantics are typically used as static prompts or post-hoc matching signals, rather than participating in the iterative mask generation process. Through systematic analysis, we show that many errors stem from semantic misalignment rather than poor mask quality. To address this issue, we propose FlowSeg, which introduces dynamic semantic guidance via a bidirectional semantic flow between intermediate decoding states and LLM-derived condition embeddings throughout the generation process. Language conditions actively guide mask refinement at each stage, while condition embeddings are progressively updated by emerging visual evidence. This design yields semantically grounded mask representations and visually aligned language conditions, enabling more reliable matching. We further incorporate a lightweight boundary-aware refinement to selectively enhance uncertain regions without perturbing confident interiors. Extensive experiments on referring expression segmentation and reasoning segmentation tasks demonstrate that FlowSeg consistently improves language-mask alignment and achieves state-of-the-art performance. Project page: https://zkzhang98.github.io/FlowSeg_page

图像分割多模态大模型语义对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。