构建大规模通用关键点检测数据集与模型，支持多模态提示。
GKDT: General Keypoint Detection Transformer
- 基于DINOv3的Transformer架构，支持视觉、文本或双模态提示。
- 在22个测试集上多数类别达90%以上[email protected]准确率。
- 适用于跨域实际场景，适合需要通用关键点识别的研究者。
随着预训练视觉与语言模型的发展，计算机视觉正从窄领域向开放域识别演进。构建一个强大且通用的关键点检测（GKD）模型以支持多样化任务变得日益重要。为此，我们首次提出一个大规模统一关键点数据集MegaKPT，包含来自29个现有数据集的超130万种多样物体实例，并具备高质量的统一标注及关键点文本描述。基于MegaKPT，我们开发了GKDT——一种基于DINOv3的简单、灵活且强大的Transformer模型。GKDT支持视觉提示、文本提示或两者结合。为提升训练效果，我们还提出混合模态提示训练和动态重要性采样等策略。在包含已见与未见物体的22个测试集上，单个GKDT模型在广泛类别上的关键点检测表现优异且泛化能力强，多数类别达到超过90%的[email protected]准确率，具有很高的实际应用价值。数据集、模型与代码将公开于https://github.com/AlanLuSun/General-Keypoint-Detection。
原文摘要 · Abstract (English)
With the emergence of various pre-trained vision and language models, computer vision is shifting from narrow-domain to open-domain recognition. The construction of a more powerful yet general keypoint detection (GKD) model to support diverse tasks has become increasingly important in the field. To this end, we firstly present a large-scale unified keypoint dataset called MegaKPT. The dataset is composed of over 1.3 million diverse object instances from twenty-nine existing datasets, and enjoys high-quality unified annotations with keypoint text descriptions. Based on MegaKPT, we develop GKDT, a simple, flexible and powerful DINOv3 based Transformer model for General Keypoint Detection. Our GKDT supports visual prompts, text prompts, or both. To enhance model training, we also propose a suite of useful strategies such as mix-modal prompted training and dynamic importance sampling. By testing over 22 test sets with seen or unseen objects, our single GKDT model shows strong performance and generality in detecting keypoints on broad categories, with most categories over 90\% [email protected] accuracy, offering high practical applicability to real-world problems. The dataset, models, and codes will be released at https://github.com/AlanLuSun/General-Keypoint-Detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。