针对视频表情识别中的个体差异,提出轻量级在线缓存方法提升模型适应能力。
Test-Time Adaptation with Online Personalized Energy-Based Cache for Fine-Grained Video Expression Recognition

- 基于能量模型从目标视频中动态生成个性化类别原型。
- 在三个数据集上超越现有方法,且计算与内存开销低。
- 适合实时视频情绪分析场景,尤其对罕见表情有更好表现。
视频面部表情识别(FER)因需捕捉个体间差异的细微、时序演变情绪状态而具挑战性。尽管视觉-语言模型提供可迁移的视觉语义表征,但基于无主体数据训练的模型在推理时遭遇个体分布偏移会性能下降。现有测试时自适应(TTA)方法通常在推理中更新模型参数,增加计算成本和延迟;缓存方法虽避免参数更新,但常需足够目标样本构建可靠类原型,这在适应初期及稀有类别下难以实现。本文提出基于能量的缓存个性化方法(EB-CaP),一种面向视频FER的主体感知在线TTA方法,能为每个目标视频生成个性化的类别原型。EB-CaP利用轻量级能量模型,从当前未标注视频中采样原型并在线填充个性化缓存,无需积累大量目标数据或存储多样源原型。其能量函数仅依赖预训练CLIP:目标视频嵌入与类别文本嵌入间的相似性指导原型采样。同时,正负缓存分别存储可信与不确定的目标嵌入。自适应熵门根据置信度分布变化控制缓存更新,多样性门限制冗余样本。最终预测融合缓存得分与当前CLIP得分。在BioVid、StressID和BAH数据集上的实验表明,EB-CaP优于现有先进TTA方法,且保持低计算与内存开销。代码已公开于https://github.com/MasoumehSharafi/EB-CaP。
原文摘要 · Abstract (English)
Facial expression recognition (FER) in videos is challenging because models must identify subtle, temporally evolving affective states that vary across individuals. Although vision-language models provide transferable visual-semantic representations, models trained on subject-independent data often degrade under subject-specific distribution shifts at inference time. Existing test-time adaptation (TTA) methods commonly update model parameters during inference, increasing computational cost and latency. Cache-based methods avoid parameter updates, but they usually require enough target samples to form reliable class prototypes, which is difficult early in adaptation and for rarely observed classes. We introduce Energy-Based Cache Personalization (EB-CaP), a subject-based online TTA method for video FER that generates class-specific prototypes personalized to each target video. EB-CaP uses a lightweight energy-based model to sample prototypes from the current unlabeled video and populate a personalized cache online, without accumulating large amounts of target data or storing diverse source prototypes. Its energy function relies only on pretrained CLIP: similarities between the target video embedding and class text embeddings guide prototype sampling. In parallel, positive and negative caches store reliable and uncertain target embeddings. An adaptive entropy gate controls cache updates according to the evolving confidence distribution, while a diversity gate limits redundant samples. Final predictions combine cache-derived scores with the current CLIP scores. Experiments on BioVid, StressID, and BAH show that EB-CaP outperforms state-of-the-art TTA methods while maintaining low computational and memory overhead. Code is available at https://github.com/MasoumehSharafi/EB-CaP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。