复杂度自适应对冻结句向量有效吗?研究发现配对级难度更有用。
When Does Complexity Conditioning Help a Frozen Sentence Embedding? A Controlled Study of Per-Sentence and Pair-Level Difficulty Adaptation
- 用配对级难度信号调制轻量适配器,而非单句复杂度
- 在STS-B和QQP上分别提升0.022和0.037的Spearman相关性
- 适合想改进冻结句向量排序性能的研究者
我们在一个多种子控制实验中检验了句向量应适应输入难度的直觉:一个轻量级后编码适配器连接到冻结的Qwen3-Embedding-0.6B编码器,仅访问其最终池化嵌入,在四个释义与语义相似性任务(PAWS、MRPC、QQP、STS-B)上评估。朴素方法失败:基于表面的单句复杂度与冻结基线误差几乎无关(皮尔逊相关≈0.05),且在常数或随机对照下无优势,反而降低饱和基线表现。即使目标与非循环的配对难度信号对齐,单句门控仍无法可靠捕捉难度,因难度主要属于配对属性而非单句。相反,由预留交叉编码器难度信号控制的小型配对级残差模块在更大、更分级的任务上持续带来收益,包括在STS-B上+0.022、在QQP上+0.037的Spearman相关性提升,同时在所有种子下保持与冻结基线一致。该有效形式作用于句子对而非单句,因此模型应理解为对缓存冻结嵌入的轻量重排序器,而非替代单向量嵌入;本文不宣称达到最先进水平。贡献在于提供了一个关于难度感知适配何时有效、何时失效的受控分析,以及一个预训练诊断工具,可预测可用提升空间。
原文摘要 · Abstract (English)
A common intuition is that sentence embeddings should adapt to the difficulty of the input. We test this intuition in a controlled, multi-seed setting: a lightweight post-encoder adapter attaches to a frozen Qwen3-Embedding-0.6B encoder, accessing only its final pooled embedding, and is evaluated on four paraphrase and semantic-similarity tasks (PAWS, MRPC, QQP, STS-B). The naive form of the idea fails: surface-based per-sentence complexity is nearly uncorrelated with frozen-baseline error (Pearson approximately 0.05) and provides no advantage over constant or shuffled controls, while degrading a saturated baseline. Even when the target is aligned to a non-circular pair-difficulty signal, the per-sentence gate still cannot reliably capture difficulty because difficulty is primarily a property of the pair, not the individual sentence. In contrast, a small pair-level residual gated by a held-out cross-encoder difficulty signal yields consistent gains on the larger and graded tasks, including +0.022 Spearman on STS-B and +0.037 on QQP, while remaining anchored to the frozen baseline across all seeds. Because this useful form operates on sentence pairs rather than individual sentences, the resulting model is best understood as a lightweight re-ranker over cached frozen embeddings, not a replacement single-vector embedding; we make no state-of-the-art claim. Our contribution is a controlled account of when difficulty-aware adaptation helps and when it fails, together with a pre-training diagnostic that predicts the available headroom.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。