让模型先找物体再对齐语言，提升儿童视角视频中词汇理解精度。
Objects Before Words: Object-First Inductive Biases for Grounding Language in Child-View Video
- 先提取候选物体，通过追踪构建轻量级物体文件
- 用原型空间多实例对比学习对齐语句与物体包
- 适合处理语音稀疏、画面杂乱的婴儿视角数据
从婴儿视角视频中学习词语的具身意义需解决两个歧义：命名对象何时出现、在复杂画面中何处。在SAYCam类数据中，照护者话语稀疏且与第一人称视频弱同步，单帧对比配对会产生噪声正样本，目标物体可能缺失或被干扰物混淆。我们提出BabyMind，一种面向稀疏、噪声监督下儿童视角对比学习的物体优先偏差。BabyMind利用离线掩码区域接口提取候选物体嵌入，通过追踪将短语中心窗口内的候选物体链接成轻量级物体文件，并使用原型空间多实例对比目标将语句对齐至物体文件集合。轨迹一致性与全局物体一致性正则化稳定了学习过程，并将物体文件结构融入评估时使用的全局帧嵌入。在SAYCam-S上，BabyMind相比CVCL提升Labeled-S 15强制选择准确率2.6个百分点，并在词汇表内分布外基准上持续取得提升。代码已开源于https://github.com/sathiiii/BabyMind。
原文摘要 · Abstract (English)
Learning grounded word meaning from natural experience requires resolving two ambiguities in infant-view recordings: when the named referent appears and where it is in a cluttered frame. In SAYCam-style data, caregiver speech is sparse and weakly synchronized with egocentric video, so single-frame contrastive pairing yields noisy positives in which the intended object is absent or entangled with distractors. We propose BabyMind, an object-first bias for child-view contrastive learning under sparse, noisy supervision. BabyMind extracts candidate object embeddings using an offline mask-based region interface, links candidates across a short utterance-centered window into lightweight object files via tracking, and aligns utterances to bags of object files with a prototype-space multiple-instance contrastive objective. Track-coherence and global-object agreement regularizers stabilize learning and transfer object-file structure into the global frame embedding used at evaluation. On SAYCam-S, BabyMind improves Labeled-S 15 forced-choice accuracy by +2.6 points over CVCL and yields consistent gains on in-vocabulary out-of-distribution benchmarks. Code is available at https://github.com/sathiiii/BabyMind.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。