将图像风格视为语义信息，分离内容与外观表征。
Stylistic-STORM (ST-STORM) : Perceiving the Semantic Nature of Appearance
- 设计双流架构，用门控机制分离内容与风格特征
- 风格分支在多天气数据上达97%准确率，仅用10%标签
- 适合自动驾驶、医学影像等需关注外观的场景
自监督学习主流范式（如MoCo、DINO）通过忽略光照、几何变化等外观扰动来获得鲁棒表征，适用于不依赖外观的物体识别。但在气象分析、自动驾驶等任务中，雨痕、雪粒、大气散射等外观特征正是关键判别信号。本文提出ST-STORM，一种混合自监督框架，将风格视为可解耦的语义模态。模型包含内容与风格双分支：内容分支采用JEPA+对比学习，对风格变化保持不变；风格分支通过特征预测与对抗约束，捕获纹理、对比度、散射等外观特征。在多个任务上验证：在ImageNet-1K上内容分支保持F1=80%，在多天气分类中风格分支达F1=97%，在ISIC 2024皮肤癌检测中使用10%标注数据即达F1=94%，显著提升关键外观信息的保留能力。
原文摘要 · Abstract (English)
One of the dominant paradigms in self-supervised learning (SSL), illustrated by MoCo or DINO, aims to produce robust representations by capturing features that are insensitive to certain image transformations such as illumination, or geometric changes. This strategy is appropriate when the objective is to recognize objects independently of their appearance. However, it becomes counterproductive as soon as appearance itself constitutes the discriminative signal. In weather analysis, for example, rain streaks, snow granularity, atmospheric scattering, as well as reflections and halos, are not noise: they carry the essential information. In critical applications such as autonomous driving, ignoring these cues is risky, since grip and visibility depend directly on ground conditions and atmospheric conditions. We introduce ST-STORM, a hybrid SSL framework that treats appearance (style) as a semantic modality to be disentangled from content. Our architecture explicitly separates two latent streams, regulated by gating mechanisms. The Content branch aims at a stable semantic representation through a JEPA scheme coupled with a contrastive objective, promoting invariance to appearance variations. In parallel, the Style branch is constrained to capture appearance signatures (textures, contrasts, scattering) through feature prediction and reconstruction under an adversarial constraint. We evaluate ST-STORM on several tasks, including object classification (ImageNet-1K), fine-grained weather characterization, and melanoma detection (ISIC 2024 Challenge). The results show that the Style branch effectively isolates complex appearance phenomena (F1=97% on Multi-Weather and F1=94% on ISIC 2024 with 10% labeled data), without degrading the semantic performance (F1=80% on ImageNet-1K) of the Content branch, and improves the preservation of critical appearance
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。