arXiv:2602.16500cs.CL2026-02被引 1

用拓扑结构优化软提示,让模型更好学且更可解释

Optimizing Soft Prompt Tuning via Structural Evolution

  • 用拓扑数据分析软提示的结构演化,量化参数间连接与冗余
  • 结构稳定紧凑的提示在下游任务上表现更优,提升性能1.5%-3.2%
  • 新损失函数加速收敛,适合追求可解释性的模型调优研究者

软提示微调利用连续嵌入捕捉大语言模型中的任务特定信息,在少样本场景下表现优异。然而,软提示依赖高维隐式表示,缺乏显式语义和可追踪的训练行为,限制了其可解释性。为此,我们提出基于拓扑形态演化的软提示优化方法。具体地,采用拓扑数据分析(TDA)中的持久同调技术,量化软提示在连续参数空间中的结构表征及其训练过程的演化。定量分析表明,拓扑结构稳定且紧凑的软提示能获得更好的下游性能。基于此实证观察,构建用于优化软提示微调的损失函数——拓扑软提示损失(TSLoss),通过量化参数间连通性和冗余性,引导模型学习结构稳定的适配。大量实验显示,使用TSLoss训练可加速收敛并提升调优性能,为从结构与拓扑视角理解与优化软提示微调提供了可解释的方法。

原文摘要 · Abstract (English)

Soft prompt tuning leverages continuous embeddings to capture task-specific information in large pre-trained language models (LLMs), achieving competitive performance in few-shot settings. However, soft prompts rely on high-dimensional, implicit representations and lack explicit semantics and traceable training behaviors, which limits their interpretability. To address this limitation, we propose a soft prompt tuning optimization method based on topological morphological evolution. Specifically, we employ persistent homology from topological data analysis (TDA) to quantify the structural representations of soft prompts in continuous parameter space and their training process evolution. Quantitative analysis shows that topologically stable and compact soft prompts achieve better downstream performance. Based on this empirical observation, we construct a loss function for optimizing soft prompt tuning, termed Topological Soft Prompt Loss (TSLoss). TSLoss guides the model to learn structurally stable adaptations by quantifying inter-parameter connectivity and redundancy. Extensive experiments show that training with TSLoss accelerates convergence and improves tuning performance, providing an interpretable method to understand and optimize soft prompt tuning from structural and topological perspectives.

软提示拓扑分析可解释性模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。