用查询匹配提升视频动作检测的时序一致性
Query matching for spatio-temporal action detection with query-based object detector
- 通过查询匹配对齐不同帧中的同一物体
- 在JHMDB21上性能显著提升
- 适合关注视频时序建模的研究者
本文将基于查询的物体检测模型DETR扩展至时空动作检测任务，该任务需保持视频中的时序一致性。方法对每一帧应用DETR，并利用特征偏移引入时序信息。然而，各帧中DETR的物体查询可能对应不同物体，简单特征偏移效果不佳。为此，提出跨帧查询匹配机制，确保同一物体的查询被正确对齐并用于特征偏移。实验表明，在JHMDB21数据集上，使用所提查询匹配进行特征偏移后，性能显著提升。
原文摘要 · Abstract (English)
In this paper, we propose a method that extends the query-based object detection model, DETR, to spatio-temporal action detection, which requires maintaining temporal consistency in videos. Our proposed method applies DETR to each frame and uses feature shift to incorporate temporal information. However, DETR's object queries in each frame may correspond to different objects, making a simple feature shift ineffective. To overcome this issue, we propose query matching across different frames, ensuring that queries for the same object are matched and used for the feature shift. Experimental results show that performance on the JHMDB21 dataset improves significantly when query features are shifted using the proposed query matching.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。