让不同触觉传感器共享感知能力，实现跨设备触觉迁移
Heterogeneous Tactile Transformer
- 用特定编码器+共享Transformer结构，学习跨传感器共通触觉表征
- 在160万对同步触觉数据上预训练，可适配新任务和未见传感器
- 适合需要多源触觉数据融合的机器人操控与感知研究者
触觉传感器具有固有异质性：在一种传感器上训练的模型无法直接用于另一种，限制了从多样化触觉数据中规模化学习丰富接触感知策略。为此，我们提出异质触觉Transformer（HTT），一种在异质传感器间学习共享触觉表征的框架。HTT包含传感器专用编码器和共享Transformer主干，通过每模态掩码重建及成对传感器间的跨模态对齐进行预训练。预训练使用我们构建的新颖异质成对触觉（HPT）数据集，包含四个基于视觉和阵列的触觉传感器间的160万组同步帧。在多种触觉感知和真实世界操控任务中，HTT被证明能学习可迁移的表征，适应新任务及此前未见的传感器。数据集、代码和模型检查点将在发表后公开于https://jxbi1010.github.io/htt-gh-page/。
原文摘要 · Abstract (English)
Tactile sensors are inherently heterogeneous: a model trained on one sensor cannot be directly used on another, which limits learning contact-rich manipulation policies from diverse tactile data at scale. To bridge this gap, we propose the Heterogeneous Tactile Transformer (HTT), a framework that learns shared tactile representations across heterogeneous sensors. HTT consists of sensor-specific encoders and a shared transformer trunk, and is pretrained with per-modality masked reconstruction together with cross-modal alignment between paired sensors. Pretraining uses our novel Heterogeneous Paired Tactile (HPT) dataset, containing 1.6M synchronized paired frames across four vision- and array-based tactile sensors. Across distinct tactile perception and real-world manipulation tasks, HTT is shown to learn transferable representations that adapt to new tasks and previously unseen sensors. Dataset, code, and model checkpoints will be released upon publication at https://jxbi1010.github.io/htt-gh-page/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。