提出新型视觉语言模型,实现超长输入下的高效推理与高精度感知。
InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
- 混合线性与稀疏注意力结构,结合门控差分网络提升效率。
- 256K上下文下预填充速度提升5倍,推理速度达25帧/秒。
- 适合需要长序列处理的实时视频理解与离线检索场景。
视觉语言模型(VLMs)正面临超长多模态理解任务的挑战。线性架构虽具恒定计算与内存开销,但高频视觉感知能力弱于标准Transformer。为此,我们提出InfiniteVL:先构建InfiniteVL-Base,将少量全连接注意力层与门控差分网络交替使用,并通过定制蒸馏与微调策略,在保持等效多模态性能的同时实现1.7倍解码加速。为突破全连接注意力的二次复杂度瓶颈,进一步提出长序列架构微调策略,将密集注意力转化为视觉专用稀疏机制,衍生出InfiniteVL-Offline(离线检索)与InfiniteVL-Online(在线流式)。前者在256K上下文下实现与Transformer相当的长度泛化能力,预填充加速达5倍;后者维持常数内存开销,达到25 FPS实时吞吐。代码与模型已开源。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) are increasingly tasked with ultra-long multimodal understanding. While linear architectures offer constant computation and memory footprints, they often struggle with high-frequency visual perception compared to standard Transformers. To bridge this gap, we introduce \textbf{InfiniteVL}. We first develop a hybrid base model called \textbf{InfiniteVL-Base} that interleaves a small fraction of Full Attention layers with Gated DeltaNet. Empowered by a tailored distillation and fine-tuning strategy, InfiniteVL-Base matches the fundamental multimodal performance of equivalent Transformers while achieving a \textbf{1.7$\times$} decoding speedup. However, the quadratic complexity of the retained Full Attention inevitably becomes an efficiency bottleneck when scaling to ultra long context. To break this barrier, we propose a novel Long-Sequence Architectural Fine-Tuning strategy that seamlessly transforms the dense attention into vision-specific sparse mechanisms. This yields two specialized variants: \textbf{InfiniteVL-Offline} for offline retrieval and \textbf{InfiniteVL-Online} for online streaming. By eliminating the computation explosion of global attention without sacrificing high-frequency visual recall, InfiniteVL-Offline achieves Transformer-level length generalization with a \textbf{5x} prefill acceleration at 256K context. Concurrently, InfiniteVL-Online delivers robust streaming perception with a constant memory footprint and a real-time throughput of \textbf{25} FPS. Code and models are available at https://github.com/hustvl/InfiniteVL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。