CanViT首次实现无需任务特定设计的主动视觉基础模型，高效感知场景。
CanViT: Toward Active-Vision Foundation Models
- 用场景相对位置编码连接视觉主干与全局记忆画布，实现高效感知
- 单次低分辨率凝视即达38.5%分割准确率，推理计算量仅为前人模型1/20
- 适合需要快速、低资源视觉理解的机器人、自动驾驶等应用
主动计算机视觉通过序列化、局部凝视实现高效且类生物感知，但缺乏可扩展的通用架构与预训练方法，导致主动视觉基础模型（AVFM）研究不足。本文提出首个任务与策略无关的AVFM——CanViT。它采用场景相对位置编码，将视网膜式视觉变压器主干与空间位姿式的全局潜在工作区（画布）关联。通过创新的画布注意力机制，实现对高容量工作记忆的高效交互。通过解耦思考（主干层）与记忆（画布层），移除画布侧自注意力和全连接层，实现快速序列推理并支持高分辨率输出。提出无标签的主动视觉预训练方案：从随机位置、缩放级别和长度的低分辨率凝视序列中重建全局DINOv3嵌入。在1320万张ImageNet-21k场景与10亿随机凝视上，仅用一台H100训练166小时完成预训练。在ADE20K分割任务中，冻结的CanViT-B仅需一次低分辨率凝视即达38.5% mIoU，优于最佳已有主动模型的27.6%，且计算量减少20倍，并超越其对应输入或计算匹配的DINOv3教师模型。增加凝视次数后，性能提升至45.9% mIoU。在ImageNet-1k分类任务中，微调后达到84.5% top-1准确率，刷新主动视觉新纪录。CanViT具备泛化至更长推演、更大场景及新策略的能力。本工作显著缩小被动与主动视觉间的差距，验证了任务与策略无关的AVFM预训练潜力。
原文摘要 · Abstract (English)
Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses, but lacks scalable general-purpose architectures and pretraining pipelines, leaving Active-Vision Foundation Models (AVFMs) underexplored. We introduce CanViT, the first task- and policy-agnostic AVFM. CanViT uses scene-relative RoPE to bind a retinotopic Vision Transformer backbone and a spatiotopic scene-wide latent workspace, the canvas. Efficient interaction with this high-capacity working memory is supported by Canvas Attention, a novel asymmetric cross-attention mechanism. We decouple thinking (backbone-level) and memory (canvas-level), eliminating canvas-side self-attention and fully-connected layers to achieve fast sequential inference and scalability to high output resolutions. We propose a label-free active vision pretraining scheme, policy-agnostic passive-to-active dense latent distillation: reconstructing scene-wide DINOv3 embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths. We pretrain CanViT-B from a random initialization on 13.2 million ImageNet-21k scenes--an order of magnitude more than previous active models--and 1 billion random glimpses, in 166 hours on a single H100. On ADE20K segmentation, a frozen CanViT-B achieves 38.5% mIoU in a single low-resolution glimpse, outperforming the best active model's 27.6% with 20x fewer inference FLOPs as well as its FLOP- or input-matched DINOv3 teacher. Given additional glimpses, CanViT-B reaches 45.9% ADE20K mIoU. On ImageNet-1k classification, CanViT-B also sets a new active-vision state of the art, with 84.5% top-1 accuracy after fine-tuning. CanViT generalizes to longer rollouts, larger scenes, and new policies. Our work narrows the wide gap between passive and active computer vision, demonstrating the potential of task- and policy-agnostic AVFM pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。