TRINE用单比特流实现多模态模型高效推理，低功耗下性能远超主流芯片。
TRINE: A Token-Aware, Runtime-Adaptive FPGA Inference Engine for Multimodal AI
- 统一多种模型为矩阵运算，动态切换计算模式以适配不同层。
- 运行时剪枝与任务调度使延迟降低22.57倍，吞吐提升最高达79%。
- 适合嵌入式多模态应用，支持整数8位量化且精度损失<2.5%。
多模态模型融合视觉、语言和图神经网络，其计算与内存模式差异大，嵌入式平台难以满足实时性要求。TRINE是一种单比特流FPGA加速器与编译器，无需重配置即可完成端到端多模态推理。将各层统一为DDMM/SDDMM/SpMM，映射至可动态切换的处理单元，支持权重/输出驻留流水线、1xCS SIMD及可路由加法树（RADT）。宽度匹配的两级top-k单元实现流式令牌剪枝；依赖感知的层卸载（DALO）跨可重构处理单元重叠独立核，维持高利用率。在Alveo U50和ZCU104上评估，相比RTX 4090延迟降低最多22.57倍，相比Jetson Orin Nano降低6.86倍（功耗20-21W）；仅令牌剪枝即带来高达7.8倍加速，DALO贡献最高79%吞吐提升。采用int8量化后，各类任务准确率下降均低于2.5%，在单一比特流下实现视觉、语言与图任务的最优延迟与能效。
原文摘要 · Abstract (English)
Multimodal stacks that mix ViTs, CNNs, GNNs, and transformer NLP strain embedded platforms because their compute/memory patterns diverge and hard real-time targets leave little slack. TRINE is a single-bitstream FPGA accelerator and compiler that executes end-to-end multimodal inference without reconfiguration. Layers are unified as DDMM/SDDMM/SpMM and mapped to a mode-switchable engine that toggles at runtime among weight/output-stationary systolic, 1xCS SIMD, and a routable adder tree (RADT) on a shared PE array. A width-matched, two-stage top-k unit enables in-stream token pruning, while dependency-aware layer offloading (DALO) overlaps independent kernels across reconfigurable processing units to sustain utilization. Evaluated on Alveo U50 and ZCU104, TRINE reduces latency by up to 22.57x vs. RTX 4090 and 6.86x vs. Jetson Orin Nano at 20-21 W; token pruning alone yields up to 7.8x on ViT-heavy pipelines, and DALO contributes up to 79% throughput improvement. With int8 quantization, accuracy drops remain <2.5% across representative tasks, delivering state-of-the-art latency and energy efficiency for unified vision, language, and graph workloads-in one bitstream.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。