用大模型动态生成语义不变的改写句，更真实评估文本嵌入模型鲁棒性。
PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs
- 评估时随机生成语义一致的改写句，避免固定测试集带来的过拟合
- 7个MTEB任务中模型性能受词元空间变化影响显著，但小模型未明显更脆弱
- 支持25种语言、20个数据集，适合关注实际应用鲁棒性的研究者
当前句子嵌入评估多依赖静态基准如大规模文本嵌入基准（MTEB）。尽管有价值，但对固定测试集的反复调优会虚高得分并掩盖真实世界鲁棒性。我们提出动态评估协议PTEB，在评估时通过大模型随机生成语义保持的改写句，并聚合多次运行结果。基于黄金标注与人工验证，我们证明大模型可生成词元多样但语义一致的改写句。在7个MTEB任务中，验证了句子编码器性能对词元空间变化敏感，即使语义不变亦然。同时发现小模型相对大模型并无明显更脆弱。结果在20个数据集与25种语言上具统计稳健性。我们倡导一种新范式：减少对预定义静态基准的依赖，转向利用评估时计算资源的动态随机评估。代码已公开。
原文摘要 · Abstract (English)
Current sentence embedding evaluations typically rely on static test beds like the Massive Text Embedding Benchmark (MTEB). While invaluable, repeated tuning on a fixed suite can inflate reported scores and obscure real-world robustness. We introduce the Paraphrasing Text Embedding Benchmark (PTEB), a dynamic protocol that stochastically generates meaning-preserving paraphrases at evaluation time and aggregates results across multiple runs. Using a cost-efficient LLM-based method grounded in gold ratings and human validation, we show that LLMs generate token-diverse but semantically preserving paraphrases. Across 7 MTEB tasks, we validate our hypothesis that the performance of sentence encoders is sensitive to changes in token space even when semantics remain fixed. We also observe that smaller models are not disproportionately affected relative to larger ones. Our results are statistically robust over multiple runs spanning 20 datasets and 25 languages. More generally, we aim to propose a new evaluation paradigm in NLP that relies less on static, pre-defined benchmarks but shifts towards dynamic, stochastic evaluation leveraging eval-time compute. We make the code to run PTEB publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。