提供工具包,量化Transformer模型对同一词在不同语境中的区分能力。
Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models

- 用跨领域同形异义词作为固定词形、变化语境的测试基底
- 通过轮廓系数评估模型各层表示空间中语义分离程度
- 专为避免标签污染和可视化偏差而设计,适合语言模型可解释性研究
Transformer语言模型在嵌入层为词类分配单一向量,但普遍认为其在后续层能根据上下文区分该词的不同用法。验证此观点需一种保持词形不变、语境与语义可控变化的构造。本文手册描述了一个开源工具包,核心是‘桥接形式’——一个在多个主题领域中重复出现、词形不变但意义不同的词。详细说明了从桥接形式定义、维基百科语料获取、实例定位、层间表征提取,到基于领域配对的轮廓系数测量及成对可视化协议的全流程设计。每一步都针对特定方法学问题(如类别标签过宽导致的语义污染、轮廓系数的多群组偏差、子词分词错位、降维图轴不可比等)进行防御性设计。本手册为方法论与实现参考,不报告或解释任何具体模型或桥接形式集的结果。工具包、完整源码及实验语料已独立归档(第9节),并附持久标识符,供后续研究引用为分析工具。
原文摘要 · Abstract (English)
A transformer language model assigns a single, context-independent vector to a word type at its embedding layer, yet is widely believed to individuate that word's occurrences by context in its later layers. Testing this belief cleanly requires a construct that holds the word form fixed while its context and intended sense vary in a controlled, labeled way. This manual documents an open toolkit built around such a construct, which we call a bridge form: a single written word that recurs, unchanged, across two or more subject domains with a different sense in each. We describe, and justify, every stage of the pipeline: the declarative specification of bridge forms and their source domains, corpus acquisition from Wikipedia, occurrence localization, layer-wise representation extraction, a domain-pairwise silhouette measurement of separation in the model's representation space, and a paired visualization protocol. Each design choice is presented together with the methodological failure mode it is meant to avoid (sense contamination from overly broad category labels, the multi-group bias of the silhouette coefficient, subword-tokenization misalignment, and axis-comparability artifacts in dimensionality-reduced plots, among others). This manuscript is a methodological and implementation reference: it does not report or interpret empirical outcomes of running the toolkit on any particular model or bridge-form set. The toolkit, its full source, and the corpora used to exercise it are archived separately (Section 9) under a persistent identifier, and are intended to be cited as an instrument by studies that use it to produce and interpret empirical results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。