不训练模型,用进化式提示实现零样本推理分割
Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Prompting
- 将提示词演化为种群,通过生成-评估-进化循环迭代优化
- 在ReasonSeg上零样本超越全监督顶尖方法,达83.6 mIoU
- 适合追求零样本、无需微调的视觉语言任务研究者
推理分割要求模型理解复杂、依赖上下文的语言查询以实现像素级定位。现有主流方法依赖监督微调(SFT)或强化学习(RL),但SFT存在灾难性遗忘与领域依赖问题,而RL常受训练不稳定和预设奖励函数限制。尽管近期无训练方法避免了训练负担,却受限于静态推理范式——通常采用单次“生成后分割”的链式流程,推理深度不足,无法自我修正语言幻觉或空间误判。本文提出新型零样本框架EVOL-SAM3,将推理分割重构为推理时的进化搜索过程。不依赖固定提示,而是维护一个提示假设种群,通过“生成-评估-演化”循环迭代优化。引入视觉竞技场实现无参考的成对锦标赛评估提示适应度,设计语义变异算子注入多样性并纠正语义错误。此外,异构竞技场模块融合几何先验与语义推理,确保最终选择鲁棒性。大量实验表明,EVOL-SAM3不仅显著优于静态基线,更在挑战性的ReasonSeg基准上以零样本设置超越全监督最先进方法,达到83.6 mIoU。代码已开源:https://github.com/AHideoKuzeA/Evol-SAM3。
原文摘要 · Abstract (English)
Reasoning Segmentation requires models to interpret complex, context-dependent linguistic queries to achieve pixel-level localization. Current dominant approaches rely heavily on Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL). However, SFT suffers from catastrophic forgetting and domain dependency, while RL is often hindered by training instability and rigid reliance on predefined reward functions. Although recent training-free methods circumvent these training burdens, they are fundamentally limited by a static inference paradigm. These methods typically rely on a single-pass "generate-then-segment" chain, which suffers from insufficient reasoning depth and lacks the capability to self-correct linguistic hallucinations or spatial misinterpretations. In this paper, we challenge these limitations and propose EVOL-SAM3, a novel zero-shot framework that reformulates reasoning segmentation as an inference-time evolutionary search process. Instead of relying on a fixed prompt, EVOL-SAM3 maintains a population of prompt hypotheses and iteratively refines them through a "Generate-Evaluate-Evolve" loop. We introduce a Visual Arena to assess prompt fitness via reference-free pairwise tournaments, and a Semantic Mutation operator to inject diversity and correct semantic errors. Furthermore, a Heterogeneous Arena module integrates geometric priors with semantic reasoning to ensure robust final selection. Extensive experiments demonstrate that EVOL-SAM3 not only substantially outperforms static baselines but also significantly surpasses fully supervised state-of-the-art methods on the challenging ReasonSeg benchmark in a zero-shot setting. The code is available at https://github.com/AHideoKuzeA/Evol-SAM3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。