用小波分层编码实现图像推理分辨率自适应，节省计算量。
WAVECLIP: Wavelet Tokenization for Adaptive-Resolution CLIP
- 用小波分解替代传统图像块嵌入，支持多分辨率输入
- 推理时从低分辨率开始，仅在必要时细化，减少计算开销
- 单模型动态调节精度与算力，适合资源受限场景
我们提出WAVECLIP，一种基于小波分层编码的统一模型，支持CLIP的自适应分辨率推理。该模型用多层级小波分解替代标准图像块嵌入，使模型能从粗到细处理图像，并自然支持同一模型内的多种分辨率输入。推理时，模型从低分辨率特征开始，仅在需要时进行细化，结合键值缓存和因果跨层级注意力机制复用已有计算，仅引入新增信息。我们在零样本分类任务上评估了WAVECLIP，发现简单的置信度门控机制可实现自适应提前退出，允许用户通过单一部署模型动态选择计算与精度的平衡。本方法仅需对冻结的CLIP教师模型进行轻量级蒸馏，即可在保持竞争力精度的同时实现显著的计算节省。
原文摘要 · Abstract (English)
We introduce WAVECLIP, a single unified model for adaptive resolution inference in CLIP, enabled by wavelet-based tokenization. WAVECLIP replaces standard patch embeddings with a multi-level wavelet decomposition, enabling the model to process images coarse to fine while naturally supporting multiple resolutions within the same model. At inference time, the model begins with low resolution tokens and refines only when needed, using key-value caching and causal cross-level attention to reuse computation, effectively introducing to the model only new information when needed. We evaluate WAVECLIP in zero-shot classification, demonstrating that a simple confidence-based gating mechanism enables adaptive early exits. This allows users to dynamically choose a compute-accuracy trade-off using a single deployed model. Our approach requires only lightweight distillation from a frozen CLIP teacher and achieves competitive accuracy with significant computational savings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。