用点序列预测让大模型直接理解像素级图像分割。
Towards Pixel-Level VLM Perception via Simple Points Prediction
- 将分割任务转为语言模型生成点坐标序列。
- 两阶段训练使点序列准确匹配真实边界，性能超越复杂设计方法。
- 无需特殊结构，普通大模型就能实现高精度空间感知。
我们提出 SimpleSeg，一种简单却高效的方法，使多模态大语言模型（MLLM）具备原生像素级感知能力。该方法将分割重构为简单的序列生成问题：模型直接在语言空间内预测描述物体边界的点序列（文本坐标）。为提升精度，引入两阶段 SF→RL 训练流程，通过基于 IoU 的强化学习奖励机制优化点序列，使其精确匹配真实轮廓。我们发现标准 MLLM 架构本身具有强大的底层感知潜力，无需专用结构即可被激活。在分割基准测试中，SimpleSeg 性能与现有复杂专用方法相当甚至更优。本工作表明，仅通过点预测即可实现精准空间理解，挑战了对辅助组件的依赖，为更统一、更强的视觉语言模型开辟新路径。
原文摘要 · Abstract (English)
We present SimpleSeg, a strikingly simple yet highly effective approach to endow Multimodal Large Language Models (MLLMs) with native pixel-level perception. Our method reframes segmentation as a simple sequence generation problem: the model directly predicts sequences of points (textual coordinates) delineating object boundaries, entirely within its language space. To achieve high fidelity, we introduce a two-stage SF$\to$RL training pipeline, where Reinforcement Learning with an IoU-based reward refines the point sequences to accurately match ground-truth contours. We find that the standard MLLM architecture possesses a strong, inherent capacity for low-level perception that can be unlocked without any specialized architecture. On segmentation benchmarks, SimpleSeg achieves performance that is comparable to, and often surpasses, methods relying on complex, task-specific designs. This work lays out that precise spatial understanding can emerge from simple point prediction, challenging the prevailing need for auxiliary components and paving the way for more unified and capable VLMs. Homepage: https://simpleseg.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。