arXiv:2601.20526cs.CV2026-01

用知识纠正错误预测,让模型更准地适应新任务。

IOTA: Corrective Knowledge-Guided Prompt Learning via Black-White Box Framework

  • 黑白盒协同:用知识纠正+数据优化双路径
  • 12个图像分类任务中少样本效果超越现有方法
  • 适合需要可解释性与高精度的任务场景

近期,将预训练模型适配到下游任务受到广泛关注。以往的参数高效微调(PET)方法将预训练模型视为黑箱,仅依赖数据驱动优化,忽视其内在先验知识,限制了模型在下游任务中的潜力。为此,我们提出一种新型黑白盒提示学习框架IOTA,融合数据驱动的黑盒模块与知识驱动的白盒模块,实现下游任务适配。具体而言,白盒模块通过对比错误预测与正确认知,提取修正性知识,并将其转化为可解释的人类提示,通过修正知识引导的提示选择策略,指导黑盒模块获得更准确预测。通过联合利用知识与数据驱动的学习信号,IOTA实现了高效的下游任务适配。在12个图像分类基准上,涵盖少样本及从易到难的适配设置,实验表明修正知识有效,且本方法优于当前最优方法。

原文摘要 · Abstract (English)

Recently, adapting pre-trained models to downstream tasks has attracted increasing interest. Previous Parameter-Efficient-Tuning (PET) methods regard the pre-trained model as an opaque Black Box model, relying purely on data-driven optimization and underutilizing their inherent prior knowledge. This oversight limits the models' potential for effective downstream task adaptation. To address these issues, we propose a novel black-whIte bOx prompT leArning framework (IOTA), which integrates a data-driven Black Box module with a knowledge-driven White Box module for downstream task adaptation. Specifically, the White Box module derives corrective knowledge by contrasting the wrong predictions with the right cognition. This knowledge is verbalized into interpretable human prompts and leveraged through a corrective knowledge-guided prompt selection strategy to guide the Black Box module toward more accurate predictions. By jointly leveraging knowledge- and data-driven learning signals, IOTA achieves effective downstream task adaptation. Experimental results on 12 image classification benchmarks under few-shot and easy-to-hard adaptation settings demonstrate the effectiveness of corrective knowledge and the superiority of our method over state-of-the-art methods.

提示学习知识引导少样本学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。