8GB嵌入式设备上实现双手操作,零拷贝感知+量化推理
Bimanual Manipulation Within an 8 GB Budget: Zero-Copy Sensing and Quantized ACT on an Entry-Level Jetson
- 通过零拷贝流媒体管道减少内存复制,释放CPU资源
- 量化后推理延迟降低至12.65毫秒,成功率仍达19/20
- 首次揭示ACT量化中变压器层不参与量化现象
基于模仿学习训练的双手操作策略通常在工作站或数据中心级GPU上评估,其在嵌入式硬件上的部署成本尚不明确。本文构建了一个完全运行在NVIDIA Jetson Orin Nano Super(8 GB)上的双臂SO-101系统,仅使用桌面级显卡(RTX 3070)进行离线训练,评估任务为柔性豆袋的抓取与放置。首先,我们建立基于NVMM缓冲区的GStreamer捕获流水线,消除三摄像头感知中的冗余主机-设备数据拷贝。出乎意料的是,传统路径在内存预算内未丢帧;零拷贝感知带来的实际收益是释放了CPU资源(单核峰值利用率从98.0%降至77.0%)和降低最差情况延迟(117.31毫秒降至101.52毫秒)。其次,我们在相同示范数据下训练了ACT与扩散策略,各自按参考预算训练(ACT为10万次梯度步,扩散策略为20万次)。结果表明,ACT在19/20次试验中达成任务成功,而扩散策略即使增加一倍步数也未能收敛到可用策略(0/10),我们认为这是由于收敛成本差异而非性能上限。第三,我们将ACT转换为TensorRT:FP16将平均推理延迟从114.02毫秒降至17.93毫秒(6.4倍加速),INT8进一步降至12.65毫秒(9.0倍加速),且在三种精度下任务成功率均保持稳定(19/20、18/20、19/20)。我们报告了两个此前未记录的发现:一是TensorRT通用的INT8校准仅量化了ResNet18主干网络,其余145个变压器层全部保留,导致模型大小仅减少0.9%但延迟再降28%;二是量化需求依赖于ACT的动作分块配置——当n_action_steps=100时可全精度运行,但在每步重预测的时间集成配置下则不可行。
原文摘要 · Abstract (English)
Bimanual manipulation policies trained with imitation learning are typically evaluated on workstation or datacenter-class GPUs, leaving the cost of deploying them on embedded hardware largely uncharacterized. We present a bimanual SO-101 system running entirely on an NVIDIA Jetson Orin Nano Super (8 GB), the entry-level tier of NVIDIA's embedded line, using a desktop GPU (RTX 3070) only for offline training, evaluated on pick-and-place of a deformable beanbag. First, we build a GStreamer capture pipeline backed by NVMM buffers that removes redundant host-device copies from three-camera sensing. Contrary to expectation, the conventional path fit the memory budget and dropped no frames; what zero-copy sensing recovers is CPU headroom (peak single-core utilization 98.0% to 77.0%) and worst-case latency (117.31 ms to 101.52 ms). Second, we train ACT and Diffusion Policy on identical demonstrations, each at its own reference budget (100k gradient steps for ACT, 200k for Diffusion Policy). ACT converges to a task-competent policy (19/20 trials) while Diffusion Policy does not converge to a usable one (0/10) even at twice the step count, which we attribute to differing convergence costs rather than an accuracy ceiling. Third, we convert ACT to TensorRT. FP16 reduces mean inference latency from 114.02 ms to 17.93 ms (6.4x) and INT8 to 12.65 ms (9.0x), with task success preserved at all three precisions (19/20, 18/20, 19/20). We report two findings not previously documented for ACT: TensorRT's general-purpose INT8 calibration quantizes the ResNet18 backbone but accepts zero of 145 transformer layers, explaining INT8's negligible size reduction over FP16 (0.9%) despite a further 28% latency gain; and the need for quantization is conditional on ACT's action-chunking configuration, feasible in full precision at n_action_steps = 100 but not at the per-step re-prediction temporal ensembling requires.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。