arXiv:2607.08493cs.LGcs.CL2026-07

通过优化集成模型多样性,更准确地反映人类主观判断的不确定性。

Ensemble Diversity Optimization for Subjective Supervision

论文配图:Ensemble Diversity Optimization for Subjective Supervision
图 1 · 摘自论文原文
  • 用可微分框架联合优化集成权重、规模和校准,提升模型可靠性。
  • 在4个主观分类任务上,交叉熵降低40%-78%,Brier分数优于基线方法。
  • 适合处理标注不一致的主观任务,如情感分析、意见分歧等场景。

主观自然语言处理任务常存在系统性标注者分歧,需建模不确定性而非忽略它。本文提出集成多样性优化(EDO),一个在预测空间中统一优化集成权重、有效基数和校准的可微分框架。EDO通过Gumbel-Softmax松弛实现集成结构与规模的端到端学习,并引入有符号多样性正则项,在验证集上调优,以控制保留或抑制分歧。该正则项防止集成坍缩,实现效用-校准权衡的可控调节。框架融合软F1代理、类别加权交叉熵以应对不平衡问题,以及可靠性加权多样性以调控组内变异性。在四个主观文本分类基准(ArMIS、ConvAbuse、HS-Brexit、MD-Agreement)上的实验表明,EDO显著改善概率校准:交叉熵降低40%-78%(相对于基线),Brier分数低于Soft-CE、Soft-MD、Top-5 Voting和WEL,同时保持竞争性F1得分,并更好匹配标注者分布。结果表明,结合有符号多样性正则项联合优化集成结构,是一种高效且模型无关的建模人类主观性的监督学习方法。

原文摘要 · Abstract (English)

Subjective NLP tasks often exhibit systematic annotator disagreement, requiring models that represent uncertainty rather than collapse it. We introduce Ensemble Diversity Optimization (EDO), a prediction-space framework that jointly optimizes ensemble weights, effective cardinality, and calibration through a unified differentiable objective. EDO learns ensemble composition and size end-to-end via Gumbel-Softmax relaxation and incorporates a signed diversity regularizer, tuned on validation data, to steer optimization toward either preserving or suppressing disagreement. This regularization prevents ensemble collapse and enables controlled navigation of the utility-calibration trade-off. The framework integrates a soft F1 surrogate, class-weighted cross-entropy to address imbalance, and reliability-weighted diversity to regulate intra-ensemble variability. Experiments on four subjective text-classification benchmarks (ArMIS, ConvAbuse, HS-Brexit, MD-Agreement) show that EDO substantially improves probabilistic calibration, reducing cross-entropy (40-78% depending on baseline) and lowering Brier scores relative to Soft-CE, Soft-MD, Top-5 Voting, and WEL, while maintaining competitive F1 and better alignment with annotator distributions. These results demonstrate that jointly optimizing ensemble structure with a signed diversity regularizer provides an efficient, model-agnostic approach for modeling human subjectivity in supervised learning.

集成学习主观任务不确定性建模校准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。