提升人体与场景接触预测精度，支持语义标签与复杂场景
DecoDINO: 3D Human-Scene Contact Prediction with Semantic Classification
- 三分支架构融合双DINOv2编码器与局部注意力机制
- 在DAMON基准上二值接触F1提升7%，几何误差减半
- 可识别物体级接触语义，适合机器人与虚拟现实应用
人体与周围物体的顶点级接触预测是实现高保真人机交互模型的关键，广泛应用于机器人、增强现实/虚拟现实及行为模拟。现有方法DECO虽能估计接触，但仅限于二值接触图，难以处理软表面、遮挡、儿童及脚部误报。本文提出DecoDINO，基于DECO框架设计三分支网络，采用两个DINOv2 ViT-g/14编码器，结合类别平衡损失权重以减少偏差，并引入补丁级交叉注意力提升局部推理能力。顶点特征经轻量MLP与softmax输出语义接触标签。尽管测试了视觉-语言模型整合文本特征，最终选用更简洁结构表现更优。在DAMON基准上，DecoDINO（i）使二值接触F1分数提升7%；（ii）几何距离误差减半；（iii）新增物体级语义标签。消融实验表明LoRA微调与双编码器是性能提升关键。DecoDINO在DAMON挑战赛的两项任务中均优于基线。代码已开源：https://github.com/DavidePasero/deco/tree/main。
原文摘要 · Abstract (English)
Accurate vertex-level contact prediction between humans and surrounding objects is a prerequisite for high fidelity human object interaction models used in robotics, AR/VR, and behavioral simulation. DECO was the first in the wild estimator for this task but is limited to binary contact maps and struggles with soft surfaces, occlusions, children, and false-positive foot contacts. We address these issues and introduce DecoDINO, a three-branch network based on DECO's framework. It uses two DINOv2 ViT-g/14 encoders, class-balanced loss weighting to reduce bias, and patch-level cross-attention for improved local reasoning. Vertex features are finally passed through a lightweight MLP with a softmax to assign semantic contact labels. We also tested a vision-language model (VLM) to integrate text features, but the simpler architecture performed better and was used instead. On the DAMON benchmark, DecoDINO (i) raises the binary-contact F1 score by 7$\%$, (ii) halves the geodesic error, and (iii) augments predictions with object-level semantic labels. Ablation studies show that LoRA fine-tuning and the dual encoders are key to these improvements. DecoDINO outperformed the challenge baseline in both tasks of the DAMON Challenge. Our code is available at https://github.com/DavidePasero/deco/tree/main.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。