将多模态模型拆解为模块化组件，实现低功耗设备上的高效离线推理。
Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devices
- 分模块部署视觉、音频、投影和语言组件至最优计算单元
- 零拷贝跨加速器传输，降低42.3%端到端能耗
- 单块2000mAh电池续航近18.8小时，适合边缘设备
大型多模态模型（LMMs）天然具有模块化结构，包含视觉与音频编码器、投影模块及语言主干。然而现有系统将其整体执行，未能充分利用现代SoC中异构加速器（如NPUs、GPUs、DSPs）的能力，导致端到端延迟升高。我们提出Nanomind——一种软硬件协同设计的推理框架，将每个LMM分解为视觉、投影、语言和音频等模块化“积木”，并分别映射至最适配的计算单元。通过令牌感知缓冲管理器（TABM），在统一内存SoC上实现跨加速器零拷贝嵌入传输，绕过CPU瓶颈。结合定制硬件、电池感知调度器及融合低比特GEMM核，Nanomind可在紧凑型电池供电原型机上实现完全离线运行。相较主流边缘框架与开发套件，其端到端能耗降低42.3%；在按需低功耗模式下，该原型机可驱动LLaVA-OneVision-Qwen2-0.5B模型配合摄像头连续运行近18.8小时，仅消耗2,000 mAh电池电量。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) are inherently modular, comprising vision and audio encoders, a projector, and a language backbone. Yet existing systems execute them monolithically, underutilizing the heterogeneous accelerators (NPUs, GPUs, DSPs) on modern SoCs and inflating end-to-end latency. We present Nanomind, a hardware-software co-design inference framework that decomposes each LMM into modular "bricks"--vision, projector, language, and audio--and maps each brick to its best-suited compute units. A Token-Aware Buffer Manager (TABM) enables zero-copy embedding transfer across accelerators on unified-memory SoCs, bypassing CPU bottlenecks. Combined with customized hardware, a battery-aware scheduler, and fused low-bit GEMM kernels, Nanomind runs entirely on a compact, battery-powered prototype that operates fully offline. Nanomind reduces end-to-end energy by 42.3% against mainstream edge frameworks and devkits; in its on-demand low-power mode, the prototype runs LLaVA-OneVision-Qwen2-0.5B with a camera for nearly 18.8 hours on a single 2,000 mAh battery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。