用多个模型模拟标注者，预测词汇语义分歧。
JuniperLiu at CoMeDi Shared Task: Models as Annotators in Lexical Semantics Disagreements
- 将不同模型视为虚拟标注者，融合连续相关度与离散标签进行聚合。
- 模型间相关度标准差比聚合标签更能预测人类标注分歧。
- 适合研究语义差异建模与自动标注评估的读者。
我们提交了CoMeDi共享任务的解决方案，旨在预测多数投票（子任务1）和标注者分歧（子任务2）。方法结合模型集成策略与基于MLP及阈值的方案，利用预训练语言模型进行训练。将各模型视为虚拟标注者，通过融合连续相关度分数与离散分类标签的聚合机制，模拟标注过程并捕捉多数意见与分歧。同时采用各向异性去除技术提升性能。实验表明该方法在子任务2上表现优异。值得注意的是，不同模型扰动下的连续相关度标准差，与人类标注分歧有更强相关性，优于聚合离散标签的指标。代码已开源：https://github.com/RyanLiut/CoMeDi_Solution。
原文摘要 · Abstract (English)
We present the results of our system for the CoMeDi Shared Task, which predicts majority votes (Subtask 1) and annotator disagreements (Subtask 2). Our approach combines model ensemble strategies with MLP-based and threshold-based methods trained on pretrained language models. Treating individual models as virtual annotators, we simulate the annotation process by designing aggregation measures that incorporate continuous relatedness scores and discrete classification labels to capture both majority and disagreement. Additionally, we employ anisotropy removal techniques to enhance performance. Experimental results demonstrate the effectiveness of our methods, particularly for Subtask 2. Notably, we find that standard deviation on continuous relatedness scores among different model manipulations correlates with human disagreement annotations compared to metrics on aggregated discrete labels. The code will be published at https://github.com/RyanLiut/CoMeDi_Solution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。