让视觉语言模型像人眼一样智能聚焦，用极少像素实现高效推理。
LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models
- 模仿人类视网膜与大脑皮层的自适应采样机制，非均匀提取关键视觉信息。
- 在仅1%像素下仍保持97%性能，比均匀采样提升38%以上准确率。
- 无需训练、即插即用，适合资源受限场景下的视觉语言模型部署。
视觉语言模型（VLMs）通常对图像全视野保持统一空间精度，即使对无信息区域也投入相同计算资源。而人类视觉具有自适应、选择性与高能效特点。为此，本文首次系统分析生物启发式视觉表征方法，提出无需训练的LLMind框架，模拟人眼中心凹编码与皮层放大效应，在严苛像素预算下实现自适应高效表征。核心思想是引入生物启发自适应采样策略（BASS），通过莫比乌斯参数化模块实现非均匀采样，同时保留全局场景结构。在此基础上，设计测试时自适应的闭环语义反馈（CSF），使感知显著性与冻结VLM中的文本信息对齐。在多种场景级与区域引导型视觉问答基准上评估，结果表明显著提升：在紧凑像素预算下，相比均匀采样，平均提升20%（VQAv2）、38%（Seed-Bench）和37%（A-OKVQA）。更惊人的是，仅使用1%、3%、5%像素时，性能分别保留82%、92%、97%的全分辨率表现。此外，该方法轻量、可即插即用，兼容现有VLM，无需架构修改。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) typically assume a uniform spatial fidelity across the entire field of view of visual inputs, dedicating equal precision to even the uninformative regions. By contrast, human vision is neither uniform nor static; it is adaptive, selective, and resource-efficient. In light of this, we present the first systematic analysis of bio-inspired visual representation methods, providing insights for more efficient and adaptive VLMs. We propose LLMind (Looking Like the Mind), a novel training-free framework that mimics foveated encoding and cortical magnification in human vision to achieve adaptive, efficient representations for VLMs under tight pixel budgets. Our key idea is to explore a Bio-inspired Adaptive Sampling Strategy (BASS), enabling a Mobius-parameterized module that performs non-uniform sampling while preserving global scene structure. On top of BASS, we introduce closed-loop semantic feedback (CSF) via test-time adaptation to align perceptual saliency with textual information from the frozen VLM. We evaluate LLMind against uniform and other sampling baselines across diverse scene-level and region-guided visual question answering benchmarks. The results show dramatic gains, with average improvements of +20% on VQAv2, +38% on Seed-Bench, and +37% on A-OKVQA compared to uniform sampling under tight pixel budgets. More surprisingly, LLMind retains up to 82%, 92%, and 97% of the full-resolution performance using only 1%, 3%, and 5% of the pixels, respectively. Moreover, LLMind is lightweight, plug-and-play, and compatible with existing VLMs without requiring architectural changes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。