解决冷启动下提示词与实例选择的依赖问题，提升模型泛化能力。
Modeling Data Diversity for Joint Instance and Verbalizer Selection in Cold-Start Scenarios
- 联合优化提示模板和实例选择，建模数据多样性
- 在8个基准上降低不确定性，提升泛化性能
- 适合无标签数据场景下的少样本学习应用
基于提示的方法利用以掩码语言建模（MLM）为目标训练的预训练语言模型（PLMs）知识；然而，这些方法对提示模板、词元化器（verbalizer）及少量样本实例选择敏感，尤其在无标签数据的冷启动设置中表现不佳。现有研究忽略了实例与词元化器间的依赖关系，即实例-标签概率取决于词元化器标记在嵌入空间中的接近程度。为此，我们提出COLDSELECT，一种联合词元化器与实例选择方法，通过建模数据多样性来解决该问题。COLDSELECT将PLM词汇表与$ h_{[MASK]} $嵌入映射到共享空间，通过降维与聚类实现高效且多样化的选择。通过最小化不确定性和最大化多样性进行优化，有效捕捉数据间关系。在8个基准上的实验表明，COLDSELECT在降低不确定性与增强泛化方面优于基线方法，显著提升冷启动场景下的词元化器与少样本实例选择效果。
原文摘要 · Abstract (English)
Prompt-based methods leverage the knowledge of pre-trained language models (PLMs) trained with a masked language modeling (MLM) objective; however, these methods are sensitive to template, verbalizer, and few-shot instance selection, particularly in cold-start settings with no labeled data. Existing studies overlook the dependency between instances and verbalizers, where instance-label probabilities depend on verbalizer token proximity in the embedding space. To address this, we propose COLDSELECT, a joint verbalizer and instance selection approach that models data diversity. COLDSELECT maps PLM vocabulary and $h_{[MASK]}$ embeddings into a shared space, applying dimensionality reduction and clustering to ensure efficient and diverse selection. By optimizing for minimal uncertainty and maximal diversity, COLDSELECT captures data relationships effectively. Experiments on eight benchmarks demonstrate COLDSELECT's superiority in reducing uncertainty and enhancing generalization, outperforming baselines in verbalizer and few-shot instance selection for cold-start scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。