在2011年老显卡上跑通现代多模态模型，实测性能极限
A Modern Multimodal Assistant on a 6 GB 2011 GPU: Stage-Validated, All-GPU CUDA Inference for Fermi
- 全GPU部署，自研加速器内核优化计算效率
- 10k上下文仍保持361 tok/s，比原版快17倍
- 适合研究硬件极限与旧设备适配的开发者
本研究在2011年NVIDIA Tesla C2075（Fermi, sm_20, 6GB）上部署了迷你版多模态助手MiniCPM-V-4.6，包含SigLIP2视觉编码器和窗口注意力融合模块（16x视觉标记压缩），采用紧凑的混合门控-增量网络骨干。第一，构建全GPU推理引擎：通过一次解量化8位权重并调用厂商SGEMM，在最后一版Fermi工具链中达到FP32峰值的64%；通过分块增量规则重写循环层，速度达顺序扫描的2.8倍；发现4位权重解码反而更慢，因Fermi的半速字节拆分。第二，视觉部分逐阶段移植至sm_20 CUDA，每阶段以本地生成参考前向结果验证（完整塔结构误差1.4e-5）；发现位置嵌入桶化在有理数平局时存在实现差异，总结出规则：索引算术中的浮点平局应调用参考运算符，避免重写。第三，长上下文暴露了传统基准隐藏的O(N²)瓶颈：朴素注意力核在10k token下预填充速率从2k token时的114 tok/s降至21 tok/s；通过修改为每头调用厂商GEMM且复用现有得分缓冲区（零额外内存），速率稳定在2k时408、10k时361 tok/s（提升17倍），并通过精确针状检索验证深度达60%的有效性；相同改写使图像编码提速6倍，降至0.93秒。系统端到端回答图像问题耗时仅1.7秒。
原文摘要 · Abstract (English)
A companion study ran a 35B mixture-of-experts model on a 2011 NVIDIA Tesla C2075 (Fermi, sm_20, 6GB) as a GPU-prefill/CPU-decode hybrid, because the 4-bit model did not fit in device memory (arXiv:2606.24031). This report keeps the hardware and asks what a model that fits can do: we deploy MiniCPM-V-4.6, a modern multimodal assistant pairing a SigLIP2 vision encoder and window-attention merger (16x visual token compression) with a compact hybrid gated-delta-net backbone, entirely on the GPU. Three results. (i) An all-GPU engine built on measured foundations: projections that dequantize 8-bit weights once and call the vendor SGEMM still in the last Fermi toolchain (64% of FP32 peak; our best hand-written GEMM hit 37%, wrongly called the ceiling); a chunked delta-rule rewrite of the recurrent layers, 2.8x faster than the sequential scan once attribution exposed one bad kernel; and a measured negative: 4-bit weights make decode slower than 8-bit here, since Fermi issues nibble-unpacking shifts at half rate. (ii) The vision side is a port with a proof obligation: we translate tower, merger, and projector to sm_20 CUDA, validating every stage against a locally generated reference forward (full tower 1.4e-5). One failure, position-embedding bucketization differing on exact rational ties, generalizes to a rule: float tie-breaking in index arithmetic is implementation-defined; call the reference operator, do not reimplement it. (iii) Long context exposes an O(N^2) wall short benchmarks hide: prefill falls from 114 tok/s at 2k tokens to 21 at 10k in a naive attention kernel; per-head vendor-GEMM calls writing into the existing score buffer (zero extra memory) restore a flat profile (408 at 2k, 361 at 10k; 17x), verified by exact needle retrieval from 60% depth. The same rewrite cuts image encoding 6x, to 0.93s. The system answers an image question end-to-end in 1.7s.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。