用文本知识提升图像模型，不改推理结构也能显著增效
TextTeacher: What Can Language Teach About Images?
- 用预训练文本编码器生成图像语义锚点，辅助视觉模型训练
- ImageNet上提升最高2.7个百分点，迁移性能平均+1.0个百分点
- 无需多模态训练，效率高且保持原模型低延迟特性
柏拉图表征假设认为，足够大的模型会收敛到共享的表征几何，即使跨模态也是如此。受此启发，我们提出：语言模型的语义知识能否高效提升视觉模型？为此，我们引入TextTeacher——一种简单辅助目标，将图像描述文本嵌入作为额外信息注入图像分类训练。TextTeacher利用现成的图像标题、预训练冻结的文本编码器和轻量投影，生成语义锚点，在训练中高效引导表征学习，同时保持推理阶段模型不变。在标准ViT主干网络下，TextTeacher在ImageNet上准确率最高提升2.7个百分点（p.p.），且在相同训练方案与算力预算下，迁移性能平均提升1.0 p.p.。其效果优于视觉知识蒸馏，在同等算力下更优或同精度下快33%。分析表明，TextTeacher充当特征空间预处理器，早期塑造深层表征，通过补充语义线索提升泛化能力。该方法开销极小，无需对目标模型进行昂贵的多模态训练，同时保留纯视觉模型的简洁性与低延迟。
原文摘要 · Abstract (English)
The platonic representation hypothesis suggests that sufficiently large models converge to a shared representation geometry, even across modalities. Motivated by this, we ask: Can the semantic knowledge of a language model efficiently improve a vision model? As an answer, we introduce TextTeacher, a simple auxiliary objective that injects text embeddings as additional information into image classification training. TextTeacher uses readily available image captions, a pre-trained and frozen text encoder, and a lightweight projection to produce semantic anchors that efficiently guide representations during training while leaving the inference-time model unchanged. On ImageNet with standard ViT backbones, TextTeacher improves accuracy by up to +2.7 percentage points (p.p.) and yields consistent transfer gains (on average +1.0 p.p.) under the same recipe and compute. It outperforms vision knowledge distillation, yielding more accuracy at a constant compute budget or similar accuracy, but 33% faster. Our analysis indicates that TextTeacher acts as a feature-space preconditioner, shaping deeper layers in the first stages of training, and aiding generalization by supplying complementary semantic cues. TextTeacher adds negligible overhead, requires no costly multimodal training of the target model and preserves the simplicity and latency of pure vision models. Project page with code and captions: https://nauen-it.de/publications/text-teacher
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。