首个跨语言风格嵌入评估基准，统一评测风格表征能力
STEB: Style Text Embedding Benchmark
- 构建涵盖7语言96个数据集的风格嵌入评测基准
- 发现语义嵌入在风格任务中表现普遍不佳
- 揭示无通用最优风格嵌入，适合多任务风格研究者
尽管语义嵌入已在大规模文本嵌入基准上得到严格评估，风格嵌入的评估仍分散在各自的任务与数据集上。为弥合这一差距，我们引入风格文本嵌入基准（STEB），一个全面开源的基准，旨在标准化风格嵌入的评估。STEB包含跨7种语言的96个数据集，覆盖作者身份验证、作者检索、人工智能文本检测、语言特征探测等应用。我们发现语义嵌入在风格任务中表现一致不佳，且不存在在所有评估任务中均占优的风格嵌入。STEB代码库已开源：https://github.com/rrivera1849/STEB。
原文摘要 · Abstract (English)
While semantic embeddings are rigorously evaluated on the Massive Text Embedding Benchmark, the evaluation of style embeddings remains fragmented, with each work relying on their own set of tasks and datasets. To bridge this gap, we introduce the Style Text Embedding Benchmark, a comprehensive open-source benchmark intended to standardize the evaluation of style embeddings. STEB encompasses 96 datasets across 7 languages, spanning applications such as authorship verification, authorship retrieval, AI-text detection, probing of linguistic features, and others. We find that semantic embeddings consistently fail in stylistic tasks, and that there is no style embedding that is universally superior across all tasks evaluated. We open-source the STEB code base at: https://github.com/rrivera1849/STEB.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。