让机器在像素层面思考并行动，实现真实世界中的智能适应。
Pixelis: Reasoning in Pixels, from Seeing to Acting
- 直接在图像视频中执行缩放、分割等操作，通过动作反馈学习。
- 在六个基准上平均提升4.08%，最高达6.03%，且工具链更短可审计。
- 适合需要物理交互和无监督自适应的视觉智能系统研究者。
多数视觉语言系统仅是静态观察者：描述像素，不采取行动，无法在数据分布变化下安全优化。这种被动性限制了可泛化的、具身的视觉智能。真正的智能应通过行动学习，而非静态描述。我们提出Pixelis，一种直接在像素空间操作的代理，使用一组紧凑的可执行操作（缩放/裁剪、分割、跟踪、OCR、时间定位），并从其后果中学习。训练分三阶段：(1) 监督微调利用思维链-动作轨迹，通过掩码模仿损失强化操作与参数标记，并辅以辅助头稳定像素参数；(2) 好奇-一致奖励微调采用双驱目标，融合预测误差好奇心与相邻步骤一致性，结合KL锚定的轻量效率先验，生成短而有效的工具链；(3) 像素测试时强化学习通过检索邻居、对完整轨迹投票而非答案，更新至高保真范例，同时用KL-EMA安全控制约束漂移。在六个公开图像与视频基准上，Pixelis平均相对提升4.08%（同一8B基线，最高+6.03%于VSI-Bench），生成更短可审计的工具链，并在测试时保持在走廊内的KL值。在像素层面行动，而非抽象标记，使多模态感知与物理世界对齐，连接视觉推理与可执行结果，实现无需外部反馈的具身自适应。
原文摘要 · Abstract (English)
Most vision-language systems are static observers: they describe pixels, do not act, and cannot safely improve under shift. This passivity limits generalizable, physically grounded visual intelligence. Learning through action, not static description, is essential beyond curated data. We present Pixelis, a pixel-space agent that operates directly on images and videos via a compact set of executable operations (zoom/crop, segment, track, OCR, temporal localization) and learns from its consequences. Pixelis trains in three phases: (1) Supervised Fine-Tuning learns a pixel-tool grammar from Chain-of-Thought-Action traces with a masked imitation loss that upweights operation/argument tokens and auxiliary heads to stabilize pixel-grounded arguments; (2) Curiosity-Coherence Reward Fine-Tuning optimizes a dual-drive objective marrying prediction-error curiosity with adjacent-step coherence and a mild efficiency prior under a KL anchor, yielding short, valid, structured toolchains; (3) Pixel Test-Time RL performs label-free adaptation by retrieving neighbors, voting over complete trajectories rather than answers, and updating toward short, high-fidelity exemplars while constraining drift with a KL-to-EMA safety control. Across six public image and video benchmarks, Pixelis yields consistent improvements: the average relative gain is +4.08% over the same 8B baseline (peaking at +6.03% on VSI-Bench), computed as (ours-baseline)/baseline, while producing shorter, auditable toolchains and maintaining in-corridor KL during test-time learning. Acting within pixels, rather than abstract tokens, grounds multimodal perception in the physical world, linking visual reasoning with actionable outcomes, and enables embodied adaptation without external feedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。