用预训练模型+机器学习融合多模态数据，提升机械臂控制精度
DML-RAM: Deep Multimodal Learning Framework for Robotic Arm Manipulation using Pre-trained Models
- 图像用预训练模型，状态数据用机器学习，最后晚融合预测动作
- 在BridgeData V2和Kuka数据集上误差分别低至0.0021和0.0028
- 模块化设计，可解释性强，适合人机协同的实时控制系统
本文提出一种新型深度多模态学习框架，用于机械臂操作。不同于传统端到端或强化学习方法，该方法采用晚融合策略：利用预训练模型处理图像序列，结合机器学习算法处理机器人状态数据，融合输出以预测连续控制动作。在BridgeData V2和Kuka数据集上，最优配置（VGG16 + Random Forest）分别达到0.0021和0.0028的均方误差，展现出优异的预测性能与鲁棒性。框架具备模块化、可解释性及实时决策能力，契合自适应、人机协同的网络物理系统目标。
原文摘要 · Abstract (English)
This paper presents a novel deep learning framework for robotic arm manipulation that integrates multimodal inputs using a late-fusion strategy. Unlike traditional end-to-end or reinforcement learning approaches, our method processes image sequences with pre-trained models and robot state data with machine learning algorithms, fusing their outputs to predict continuous action values for control. Evaluated on BridgeData V2 and Kuka datasets, the best configuration (VGG16 + Random Forest) achieved MSEs of 0.0021 and 0.0028, respectively, demonstrating strong predictive performance and robustness. The framework supports modularity, interpretability, and real-time decision-making, aligning with the goals of adaptive, human-in-the-loop cyber-physical systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。