对比两种众包标注方法，发现强制选择更利于多样标注。
On Crowdsourcing Task Design for Discourse Relation Annotation
- 用预设选项强制选择连接词，提升标注多样性。
- 13万+标注对比显示自由选择导致标签趋同。
- 任务设计影响标注质量，适合研究话语关系建模者。
理解隐含话语关系需要复杂推理，需融合语义线索与背景知识，因缺乏显式连接词（如because、then）。这类关系常允许多种解释，宜以分布形式表示。本研究对比两种众包标注英语隐含话语关系的方法：自由选择法（允许标注者任意选择连接词）与强制选择法（在预设选项中选择）。我们对整个DiscoGeM 1.0语料库（初始采用自由选择法标注）重新使用强制选择法进行标注。自由选择法虽灵活直观，但13万余条标注分析显示其结果多样性不足，常集中于常见标签。结果揭示了任务设计与标注者理解能力间的相互作用。
原文摘要 · Abstract (English)
Interpreting implicit discourse relations involves complex reasoning, requiring the integration of semantic cues with background knowledge, as overt connectives like because or then are absent. These relations often allow multiple interpretations, best represented as distributions. In this study, we compare two established methods that crowdsource English implicit discourse relation annotation by connective insertion: a free-choice approach, which allows annotators to select any suitable connective, and a forced-choice approach, which asks them to select among a set of predefined options. Specifically, we re-annotate the whole DiscoGeM 1.0 corpus -- initially annotated with the free-choice method -- using the forced-choice approach. The free-choice approach allows for flexible and intuitive insertion of various connectives, which are context-dependent. Comparison among over 130,000 annotations, however, shows that the free-choice strategy produces less diverse annotations, often converging on common labels. Analysis of the results reveals the interplay between task design and the annotators' abilities to interpret and produce discourse relations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。