在ARM64设备上用软件实现张量虚拟化，突破边缘AI推理的内存瓶颈。
Bare-Metal Tensor Virtualization: Overcoming the Memory Wall in Edge-AI Inference on ARM64
- 用mmap直接映射内存，手写NEON指令优化数据访问
- 110万参数模型在M2上达60+词/秒稳定吞吐
- 开源可移植，适合研究ARM芯片内存瓶颈的开发者
大型语言模型在边缘设备上的部署受制于“内存墙”——数据移动延迟超过计算吞吐。传统推理运行时因高层抽象、动态分发和非对齐内存访问带来显著开销。本文提出一种基于软件的“虚拟张量核心”架构，专为ARM64微架构（如Apple Silicon）优化。通过绕过标准库容器，采用直接内存映射（mmap）并实现手工调优的NEON SIMD内核，实现类“软件定义直接内存访问（DMA）”。所提张量虚拟化布局（TVL）确保权重矩阵100%缓存行利用率，零拷贝加载消除初始化延迟。在110M参数模型上，M2硬件实现>60词/秒稳定吞吐。尽管专用加速器（如Apple AMX）可获更高峰值性能，本方案提供完全开源、可移植且确定性的参考实现，无需隐式依赖即可满足200ms心理语言学延迟阈值。
原文摘要 · Abstract (English)
The deployment of Large Language Models (LLMs) on edge devices is fundamentally constrained by the "Memory Wall" the bottleneck where data movement latency outstrips arithmetic throughput. Standard inference runtimes often incur significant overhead through high-level abstractions, dynamic dispatch, and unaligned memory access patterns. In this work, we present a novel "Virtual Tensor Core" architecture implemented in software, optimized specifically for ARM64 microarchitectures (Apple Silicon). By bypassing standard library containers in favor of direct memory mapping (mmap) and implementing hand-tuned NEON SIMD kernels, we achieve a form of "Software-Defined Direct Memory Access (DMA)." Our proposed Tensor Virtualization Layout (TVL) guarantees 100% cache line utilization for weight matrices, while our zero-copy loader eliminates initialization latency. Experimental results on a 110M parameter model demonstrate a stable throughput of >60 tokens/second on M2 hardware. While proprietary hardware accelerators (e.g., Apple AMX) can achieve higher peak throughput, our architecture provides a fully open, portable, and deterministic reference implementation for studying the memory bottleneck on general-purpose ARM silicon, meeting the 200ms psycholinguistic latency threshold without opaque dependencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。