只修复对任务关键的图像特征，提升恢复效率与效果
TaskTok: Delving into Task Tokens for Task-driven Image Restoration
- 用可学习开关筛选关键视觉令牌进行修复
- 在分类/分割/检测任务上性能显著提升，计算量低
- 适合需要高效图像修复的下游视觉任务应用
传统图像修复侧重感知质量，而任务驱动图像修复（TDIR）旨在最大化下游高层视觉任务的性能。现有基于生成先验的方法虽有潜力，但通常存在计算效率低、对所有潜在令牌无差别更新导致语义改变的问题。本文发现，任务相关线索在潜在令牌序列中分布不均，呈现索引特异性。据此提出TaskTok框架，通过可学习令牌开关和轻量级修复模块，仅选择性地优化任务相关的令牌。在图像分类、语义分割和目标检测任务上的大量实验表明，TaskTok在保持高计算效率的同时显著提升任务性能。源代码已公开于https://github.com/jimmy9704/TaskTok。
原文摘要 · Abstract (English)
While traditional image restoration focuses on perceptual quality, Task-Driven Image Restoration (TDIR) aims to maximize the performance of downstream high-level vision tasks. Recent approaches leveraging generative priors have shown promise for TDIR; however, they typically suffer from computational inefficiency and potential semantic alteration by indiscriminately updating all latent tokens. In this paper, we posit that not all visual information is equally important for machine perception. Through an analysis of the latent token space, we observe that task-relevant cues are unevenly distributed across the token sequence, exhibiting index-wise specialization. This suggests that selectively refining a subset of tokens can be sufficient for task-driven objectives. Leveraging this insight, we propose TaskTok, a novel framework that selectively restores only task-relevant tokens via a learnable token switch and a lightweight token refinement module. Extensive experiments across image classification, semantic segmentation, and object detection demonstrate that TaskTok significantly enhances task performance with high computational efficiency. The source code is available at https://github.com/jimmy9704/TaskTok
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。