arXiv:2606.06994cs.CLcs.DB2026-06

揭示句子编码器生成概念表示的四大原则,指导模型优化。

Principles of Concept Representation in Sentence Encoders

  • 从语义算子的低失真实现角度定义概念表征质量
  • 430万词对训练下发现结构化限制与校准机制
  • 适合关注语义几何与监督设计的研究者

为何某些句子编码器能生成优质概念表示?我们从表征组合性视角出发:只有当潜在空间能以低失真实现对应语义算子时,编码器才能支持某类概念。该框架预测了现有编码器的成功与结构性不匹配之处。通过在330万条来自WordNet和Wiktionary的同义词与定义对上进行受控消融实验,评估三个去污染数据集与修饰词标注名词短语基准,我们总结出四个原则:微调重校潜空间几何而非扩展(P1);语义信号集中在最终Transformer层,跨层聚合冗余(P2);硬负样本提升判别力并压力测试鲁棒性,但不影响检索排名,表明校准与排序可独立处理(P3);监督有效性取决于目标概念的组合类型:外延训练有助于交集型与子集型概念,却损害关系型与内涵型概念,暴露当前训练范式的结构性局限(P4)。我们发布了两个新评估数据集:DBpedia语义鸿沟基准与修饰词标注名词短语改写套件。

原文摘要 · Abstract (English)

What makes a sentence encoder produce good concept representations? We approach this through the lens of representational compositionality: an encoder supports a concept family only when its latent space admits a low-distortion realization of the corresponding semantic operator. This framing predicts both where current encoders succeed and where they are structurally mismatched to their supervision. Through a controlled ablation over encoder conditions trained on 3.3 million synonym and definition pairs from WordNet and Wiktionary, evaluated on three decontaminated splits and a modifier-labeled noun-phrase benchmark, we identify four principles. Fine-tuning recalibrates the latent geometry rather than expanding it (P1). Semantic signal concentrates in the final transformer layer before concept-specific training begins, making cross-layer pooling redundant (P2). Hard negatives improve discrimination and stress-test robustness without improving retrieval ranking, showing that calibration and ranking are independently addressable (P3). Finally, the effectiveness of supervision depends on the composition type of the target concept. Extensional training helps intersective and subsective families while degrading relational and intensional ones, exposing a structural limitation of current training paradigms (P4). We release two new evaluation datasets: a DBpedia semantic-gap benchmark and a modifier-labeled NP paraphrase suite.

句子编码器概念表征语义组合性模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。