无监督学习视频状态追踪，通过时间自蒸馏提升精度
Temporal Self-Distillation: Learning Visual State Tracking in Videos Without Supervision
- 用密集采样视频作教师，稀疏采样学生自蒸馏学习状态
- 在多个数据集上提升1.74到7.95点准确率，无需标签或额外模型
- 适合做自监督视频理解、无标注视频分析的开发者参考
我们提出S$^3$T（时间自蒸馏），据知是首个完全自包含的连续视频状态追踪框架。该方法将时间采样密度视为优势信息，基于假设：同一片段更密集的采样能更准确恢复运行状态。以此作为教师模型，与共享权重的稀疏采样学生模型匹配下一标记分布。模型自生成目标，训练无需标签、独立教师或奖励信号，且不增加推理开销。在LLaVA-OneVision-2-8B上，S$^3$T单模型提升VSTAT准确率+1.74，集成后达+2.38，结合视觉编码器适配进一步提升至+2.70；先前自演化方法对状态追踪影响甚微。从无标签合成片段学到的能力可迁移至真实视频，在VSTAT-YouTube状态追踪问题上提升+7.95，在MVBench Action Count上提升+4.50。
原文摘要 · Abstract (English)
We introduce S$^3$T (Self-Supervised Self-Distillation over Time), which, to the best of our knowledge, is the first fully self-contained framework for continuous video state tracking. Our method treats temporal sampling density as privileged information, based on the hypothesis that a denser view of the same clip recovers the running state more accurately. This view serves as the teacher, while a sparse-view student with the same weights learns to match its next-token distribution. The model generates its own target, so training requires no labels, separate teacher, or reward signal, and adds no inference cost. On LLaVA-OneVision-2-8B, S$^3$T improves VSTAT accuracy by $+1.74$ as a single model, $+2.38$ with souping, and $+2.70$ with additional vision-encoder adaptation, while prior self-evolving methods leave state tracking largely unchanged. The capability learned from unlabeled synthetic clips transfers to real videos, improving performance by $+7.95$ on VSTAT-YouTube state-tracking questions and $+4.50$ on MVBench Action Count.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。