模仿人眼视觉机制，用局部高分辨率提升视觉模型效率
FOVI: A biologically-inspired foveated interface for deep vision models
- 构建类视网膜-视觉皮层结构，将变分辨率输入转为统一密度特征图
- 在图像分类任务中仅用1/8像素量级实现媲美全分辨率模型性能
- 适合需要低功耗实时处理的可穿戴设备与机器人视觉系统
人类视觉具有中心高分辨率、周边低分辨率的聚焦特性，体现主动感知中的高效权衡——通过眼球运动将不同区域带入焦点，同时保持整体上下文。相比之下，多数计算机视觉系统对整个画面采用均匀分辨率，导致处理全分辨率图像时计算开销巨大。本文提出一种受生物视觉启发的聚焦式视觉接口（FOVI），基于视网膜与初级视觉皮层（V1）结构，将变分辨率的类视网膜传感器阵列重构为均匀密集的V1型传感器流形。感受野定义为该流形上的k近邻（kNN），并引入新型核映射技术实现kNN卷积。我们验证了两种应用场景：(1) 端到端的kNN卷积架构；(2) 对DINOv3 ViT基础模型进行聚焦式适配，结合低秩适配（LoRA）。这些模型在仅使用全分辨率基线约1/8像素量级的情况下，仍能取得相当的性能表现，为高分辨率第一人称视觉的高效可扩展主动感知提供了新路径。代码与预训练模型已公开。
原文摘要 · Abstract (English)
Human vision is foveated, with variable resolution peaking at the center of a large field of view; this reflects an efficient trade-off for active sensing, allowing eye-movements to bring different parts of the world into focus with other parts of the world in context. In contrast, most computer vision systems encode the visual world at a uniform resolution, raising challenges for processing full-field high-resolution images efficiently. We propose a foveated vision interface (FOVI) based on the human retina and primary visual cortex (V1), that reformats a variable-resolution retina-like sensor array into a uniformly dense, V1-like sensor manifold. Receptive fields are defined as k-nearest-neighborhoods (kNNs) on the sensor manifold, enabling kNN-convolution via a novel kernel mapping technique. We demonstrate two use cases: (1) an end-to-end kNN-convolutional architecture, and (2) a foveated adaptation of the DINOv3 ViT foundation model, leveraging low-rank adaptation (LoRA). These models provide competitive performance with a fraction of the pixels and computational cost of full resolution non-foveated baselines, opening pathways for efficient and scalable active sensing for high-resolution egocentric vision. Code (https://github.com/nblauch/fovi) and pre-trained models (https://huggingface.co/fovi-pytorch) are available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。