让机器视觉像人一样主动选择关注点,大幅降低计算成本。
Emulating Human-like Adaptive Vision for Efficient and Flexible Machine Visual Perception
- 将视觉感知变为逐步聚焦的决策过程,动态选择关键区域
- 在17个任务中实现最高28倍推理成本降低,准确率不变
- 无需额外标注就能训练,适合研究视觉认知与实际应用
人类视觉具有高度适应性,通过依次注视任务相关区域高效感知复杂环境。现有机器视觉模型被动处理整个场景,导致资源消耗随输入分辨率和模型规模线性增长,严重制约发展与应用。本文提出AdaptiveNN框架,推动视觉模型从‘被动’向‘主动、自适应’转变。AdaptiveNN将视觉感知建模为粗到细的序列决策过程,逐步识别并关注任务相关区域,累积多注视信息,并在满足条件时主动结束观察。我们建立融合表征学习与自奖励强化学习的理论,实现非可微AdaptiveNN的端到端训练,无需额外注视位置监督。在涵盖9类任务的17个基准上评估,AdaptiveNN实现最高28倍推理成本降低,保持精度,灵活适应不同任务需求与资源限制,且通过注视模式提升可解释性。其行为在多数情况下接近人类,具备研究视觉认知的潜力。代码已开源。
原文摘要 · Abstract (English)
Human vision is highly adaptive, efficiently sampling intricate environments by sequentially fixating on task-relevant regions. In contrast, prevailing machine vision models passively process entire scenes at once, resulting in excessive resource demands scaling with spatial-temporal input resolution and model size, yielding critical limitations impeding both future advancements and real-world application. Here we introduce AdaptiveNN, a general framework aiming to drive a paradigm shift from 'passive' to 'active, adaptive' vision models. AdaptiveNN formulates visual perception as a coarse-to-fine sequential decision-making process, progressively identifying and attending to regions pertinent to the task, incrementally combining information across fixations, and actively concluding observation when sufficient. We establish a theory integrating representation learning with self-rewarding reinforcement learning, enabling end-to-end training of the non-differentiable AdaptiveNN without additional supervision on fixation locations. We assess AdaptiveNN on 17 benchmarks spanning 9 tasks, including large-scale visual recognition, fine-grained discrimination, visual search, processing images from real driving and medical scenarios, language-driven embodied AI, and side-by-side comparisons with humans. AdaptiveNN achieves up to 28x inference cost reduction without sacrificing accuracy, flexibly adapts to varying task demands and resource budgets without retraining, and provides enhanced interpretability via its fixation patterns, demonstrating a promising avenue toward efficient, flexible, and interpretable computer vision. Furthermore, AdaptiveNN exhibits closely human-like perceptual behaviors in many cases, revealing its potential as a valuable tool for investigating visual cognition. Code is available at https://github.com/LeapLabTHU/AdaptiveNN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。