arXiv:2603.05276cs.LGcs.AI2026-03

用视觉提示让冻结的OCR模型自动优化输入,提升识别准确率。

Whispering to a Blackbox: Bootstrapping Frozen OCR with Visual Prompts

  • 通过扩散模型生成视觉提示,动态优化输入图像以适配冻结的OCR模型。
  • 在30万张退化合成文本图像上,字符错误率降低8%(相对10.6%),优于传统增强方法。
  • 无需修改模型权重,适合部署在无法更新的预训练系统中使用。

在现代机器学习中,冻结的预训练模型虽稳定高效,但常因数据分布不匹配导致性能下降。本文提出Whisperer,一种基于扩散模型的视觉提示框架,通过在像素空间中学习预处理器来适应输入,相当于‘低声提示’冻结的下游模型(如EasyOCR)。该方法将过程建模为对随机发现改进策略的行为克隆,使字符错误率(CER)在包含30万张退化合成文本图像的挑战性数据集上降低8%(相对提升10.6%),超越了手写工程基线如CLAHE。关键创新在于四阶段训练课程:利用部分训练的扩散模型进行随机探索,采样偶然改善结果,并训练模型复现这些“幸运”输出。此自举式流程仅需60 GPU小时,样本效率高,避免传统强化学习的缺陷。核心思想并非直接强化学习,而是行为克隆探索策略——随机采样中间扩散输出,筛选出能降低CER的样本,再训练模型重现。最终实现不修改模型权重的情况下,通过优化输入显著提升分类器性能。

原文摘要 · Abstract (English)

In the landscape of modern machine learning, frozen pre-trained models provide stability and efficiency but often underperform on specific tasks due to mismatched data distributions. This paper introduces the Whisperer, a novel visual prompting framework that learns diffusion-based preprocessors to adapt inputs in pixel space, effectively "whispering" enhancements to frozen downstream models like EasyOCR. By framing the process as behavioral cloning of stochastically discovered improvement policies, our method achieves an 8% absolute (10.6% relative) reduction in Character Error Rate (CER) on a challenging dataset of 300k degraded synthetic text images, surpassing hand-engineered baselines such as CLAHE. The key innovation is a four-stage training curriculum that uses behavioral cloning to amplify "lucky" improvements discovered through the stochastic exploration of a partially trained diffusion model. This approach is highly sample-efficient and avoids the pitfalls of traditional reinforcement learning. Crucially, we frame this not as naive reinforcement learning, but as behavioral cloning of an exploration policy: we stochastically sample intermediate diffusion outputs, select those that improve CER by chance, and then train the model to reproduce them. This bootstrapping curriculum (4 stages over 60 GPU-hours) amplifies random successes into a systematic strategy. In summary, by whispering to the frozen OCR through its inputs, we improve an imperfect classifier without touching its weights.

视觉提示扩散模型OCR优化冻结模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。