arXiv:2603.13786cs.CLcs.NE2026-03

不依赖投影的进化策略,高效搜索连续提示并提升少样本性能。

Projection-Free Evolution Strategies for Continuous Prompt Search

  • 直接在全维度提示空间优化,避免随机投影带来的结构损失。
  • 在GLUE七个任务上显著优于现有基线,少样本场景下效果更优。
  • 引入置信度正则化,增强模型对目标词元的判断可靠性。

连续提示搜索为自然语言处理任务提供了比传统参数调优更高效的替代方案。然而,其实际效果常受限于目标函数的黑箱性质及高维特性。现有方法通常通过将搜索限制在随机投影的低维子空间来缓解问题,但投影机制的有效性与动机尚不明确。本文首次实证表明,尽管提示空间具有低维结构,随机投影仍无法充分捕捉该结构。基于此,我们提出一种无需投影的进化策略提示搜索方法,通过在全提示空间中优化并结合内在维度自适应机制,实现媲美甚至超越现有方法的搜索能力,且无额外计算开销。此外,为缩小少样本场景下的泛化差距,我们引入基于置信度的正则化机制,系统性提升模型对目标词元的置信度。在来自GLUE基准的七个自然语言理解任务上的实验结果表明,所提方法显著优于现有基线。

原文摘要 · Abstract (English)

Continuous prompt search offers a computationally efficient alternative to conventional parameter tuning in natural language processing tasks. Nevertheless, its practical effectiveness can be significantly hindered by the black-box nature and the inherent high-dimensionality of the objective landscapes. Existing methods typically mitigate these challenges by restricting the search to a randomly projected low-dimensional subspace. However, the effectiveness and underlying motivation of the projection mechanism remain ambiguous. In this paper, we first empirically demonstrate that despite the prompt space possessing a low-dimensional structure, random projections fail to adequately capture this essential structure. Motivated by this finding, we propose a projection-free prompt search method based on evolutionary strategies. By directly optimizing in the full prompt space with an adaptation mechanism calibrated to the intrinsic dimension, our method achieves competitive search capabilities without additional computational overhead. Furthermore, to bridge the generalization gap in few-shot scenarios, we introduce a confidence-based regularization mechanism that systematically enhances the model's confidence in the target verbalizers. Experimental results on seven natural language understanding tasks from the GLUE benchmark demonstrate that our proposed approach significantly outperforms existing baselines.

提示搜索进化策略少样本学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。