发现大模型中存在视觉核心层，仅调优25%分布层即可保留99%视觉性能。
Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference
- 通过分析人类视觉认知机制，定位大模型中类似视觉区域的层分布。
- 仅更新25%稀疏均匀分布的层，保持99%视觉性能且训练提速。
- 提出基于视觉区域的剪枝新范式，可大幅减少计算量适合部署场景。
大型视觉语言模型（LVLMs）通常通过视觉指令微调来学习视觉能力，涉及投影层和语言模型骨干的联合更新。受人脑视觉区域概念启发，我们探究了大模型中是否存在类似的视觉区域作为认知核心，并研究通过选择性层调优实现高效训练的可行性。以Bunny-Llama-3-8B-V为分析对象，另用三个其他LVLM在多种视觉与文本任务上验证，结果表明：当仅对25%稀疏且均匀分布的层进行更新时，可保持近99%的视觉性能，同时维持或提升文本任务表现，显著降低训练时间。基于此策略，进一步提出一种基于视觉区域的剪枝范式，移除视觉区域外非关键层，实现极小性能损失。该研究为不同模型提供了高效、有效的视觉语言模型训练与推理策略。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) typically learn visual capacity through visual instruction tuning, involving updates to both a projector and their LLM backbones. Inspired by the concept of a visual region in the human brain, we investigate the existence of an analogous \textit{visual region} within LLMs that functions as a cognitive core, and explore the potential of efficient training of LVLMs via selective layers tuning. Using Bunny-Llama-3-8B-V for detailed analysis and other three LVLMs for validation across diverse visual and textual tasks, we find that selectively updating 25\% of LLMs layers, when sparsely and uniformly distributed, can preserve nearly 99\% of visual performance and maintain or improve textual task results, while effectively reducing training time. Based on this targeted training approach, we further propose a novel visual region-based pruning paradigm, removing non-critical layers outside the visual region, which can achieve minimal performance loss. This study offers an effective and efficient strategy for LVLM training and inference by activating a layer-wise visual region within LLMs, which proves consistently effective across different models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。