NVILA通过先扩容后压缩，让视觉语言模型更高效且保持高精度。
NVILA: Efficient Frontier Visual Language Models
- 先提升分辨率再压缩视觉令牌，提升处理能力
- 训练成本降低1.9-5.1倍，推理延迟减少1.2-2.8倍
- 适合追求高效部署的开发者与研究者使用
视觉语言模型（VLMs）近年来在准确性上取得了显著进展，但效率问题仍被忽视。本文提出NVILA，一组开源的VLM，旨在同时优化效率与准确率。基于VILA架构，我们首先提升空间和时间分辨率，再对视觉令牌进行压缩，采用“先扩容后压缩”策略，使NVILA能高效处理高分辨率图像与长视频。我们还系统性地优化了模型全生命周期的效率，涵盖训练、微调到部署阶段。NVILA在多个图像与视频基准测试中达到或超过主流开源及专有VLM的性能，同时将训练成本降低1.9–5.1倍，预填充延迟降低1.6–2.2倍，解码延迟降低1.2–2.8倍。代码与模型已公开，以促进可复现性。
原文摘要 · Abstract (English)
Visual language models (VLMs) have made significant advances in accuracy in recent years. However, their efficiency has received much less attention. This paper introduces NVILA, a family of open VLMs designed to jointly optimize efficiency and accuracy. Building on top of VILA, we improve its model architecture by first scaling up the spatial and temporal resolutions, and then compressing visual tokens. This "scale-then-compress" approach enables NVILA to efficiently process high-resolution images and long videos. We further conduct a systematic investigation that enhances NVILA's efficiency throughout its entire lifecycle, from training and fine-tuning to deployment. NVILA matches or surpasses the accuracy of leading open and proprietary VLMs across a wide range of image and video benchmarks. At the same time, it reduces training cost by 1.9-5.1x, prefilling latency by 1.6-2.2x, and decoding latency by 1.2-2.8x. We release our code and models to facilitate reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。