用文本指令替代图像,大幅降低视觉分类推理成本。
Instruction Distillation: Text Instructions as Visual Examples

- 为每张训练图生成结构化文本指令,保留类内多样性。
- 在K=5时减少2.9倍上下文令牌和3.3倍延迟。
- 图文结合更优,文本提供逻辑规则,图像提供视觉模式。
基于多模态大模型的视觉上下文学习在细粒度图像分类中表现优异,但每次检索图像会消耗数百个上下文令牌,导致大规模K值推理代价高昂。本文提出指令蒸馏:一种离线过程,由多模态大模型为每张训练图像生成结构化识别指令,包含通用外观特征、区分相似类别的关键特征及常见混淆点。与以往按类别生成单一描述不同,该方法按图像生成,保留了类内视觉多样性。推理时,使用单一CLIP检索索引测试五种配置:零样本、图像ICL、仅指令ICL及两种混合变体(检索邻居按图像与指令分配)。在七个细粒度基准和两种多模态大模型架构下,指令驱动方案在K=1时性能持平或超越图像ICL,K=5时上下文令牌减少2.9倍,推理延迟降低3.3倍。混合配置表明视觉与文本信号互补:图像提供可观察模式,指令提供明确规则,两者结合显著提升上下文质量,性能更优。
原文摘要 · Abstract (English)
Visual in-context learning (ICL) with multimodal large language models (MLLMs) is effective for fine-grained visual classification, but each retrieved image example consumes several hundred context tokens, making large-$K$ settings prohibitively expensive at inference scale. We propose Instruction Distillation: an offline procedure in which the MLLM itself generates, for each individual training image, a structured identification instruction encoding general appearance cues, features that differentiate the class from visually similar ones, and a common confusion point. Unlike prior work that produces a single description per class, our instructions are generated per training image, preserving the intra-class visual diversity that per-class descriptions collapse. At inference time, we study five configurations sharing a single CLIP retrieval index: zero-shot, image ICL, instruction-only ICL, and two hybrid variants in which retrieved neighbors are split between images and instructions. Across seven fine-grained benchmarks and two MLLM backbones, instruction based pipelines match, or exceeds image ICL at $K{=}1$ and reduces per-query tokens by $2.9\times$ and inference latency by $3.3\times$ at $K{=}5$. Hybrid configurations further show that visual and textual ICL signals are complementary, images give visual patterns to learn and see, while instructions give explicit rules and logic. When both of these are provided, the quality of context improves, which is noticeable in the performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。