用可控制的文本生成评估嵌入模型，让语义变化更清晰。
Sentence Smith: Controllable Edits for Evaluating Text Embeddings
- 将句子解析为语义图，再通过人工规则修改并重生成。
- 生成的文本能精准区分不同语义变化，适合细粒度评估。
- 只需小模型，资源高效，适合研究者快速构建测试数据。
可控且透明的文本生成是NLP长期目标。传统方法因解析与生成能力不足受限。本文提出Sentence Smith框架：1. 将句子解析为语义图；2. 应用人工设计的语义操作规则；3. 从修改后的图生成新文本；4. 通过蕴含检验验证变换有效性。利用该框架，我们生成了挑战文本嵌入模型的难负样本对。由于生成过程可控，可清晰分离各类语义偏移，实现细粒度评估，解决现有基准中语言现象不透明的问题。人工验证显示生成文本质量良好。尤其值得注意的是，该方法仅依赖小型神经网络，资源开销极低。
原文摘要 · Abstract (English)
Controllable and transparent text generation has been a long-standing goal in NLP. Almost as long-standing is a general idea for addressing this challenge: Parsing text to a symbolic representation, and generating from it. However, earlier approaches were hindered by parsing and generation insufficiencies. Using modern parsers and a safety supervision mechanism, we show how close current methods come to this goal. Concretely, we propose the Sentence Smith framework for English, which has three steps: 1. Parsing a sentence into a semantic graph. 2. Applying human-designed semantic manipulation rules. 3. Generating text from the manipulated graph. A final entailment check (4.) verifies the validity of the applied transformation. To demonstrate our framework's utility, we use it to induce hard negative text pairs that challenge text embedding models. Since the controllable generation makes it possible to clearly isolate different types of semantic shifts, we can evaluate text embedding models in a fine-grained way, also addressing an issue in current benchmarking where linguistic phenomena remain opaque. Human validation confirms that our transparent generation process produces texts of good quality. Notably, our way of generation is very resource-efficient, since it relies only on smaller neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。