用拓扑方法揭示模型如何处理文本标注分歧中的模糊性。
When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguity
- 用拓扑工具Mapper分析文本嵌入空间结构
- 98%连通分量预测纯度超90%，但标签匹配率下降
- 适合研究模型对主观任务的决策机制
语言模型常以准确率等标量指标评估，但这类指标无法捕捉模型内部对模糊性的表征，尤其在人工标注存在分歧时。本文提出一种拓扑视角，分析微调模型如何编码模糊性与一般实例。在MD-Offense数据集上对RoBERTa-Large的应用显示，微调使嵌入空间重构为模块化、非凸区域，与模型预测对齐，即使在高度模糊案例中亦然。超过98%的连通分量具有≥90%的预测纯度，但与真实标签的对齐度在模糊数据中下降，暴露出结构置信与标签不确定性之间的隐含张力。相比PCA或UMAP等传统工具，Mapper直接捕获几何结构，揭示决策区域、边界坍缩和过度自信聚类。研究结果表明，Mapper是理解模型化解模糊性的有力诊断工具。除可视化外，还可提供拓扑度量，或用于主观自然语言处理任务的主动建模策略。
原文摘要 · Abstract (English)
Language models are often evaluated with scalar metrics like accuracy, but such measures fail to capture how models internally represent ambiguity, especially when human annotators disagree. We propose a topological perspective to analyze how fine-tuned models encode ambiguity and more generally instances. Applied to RoBERTa-Large on the MD-Offense dataset, Mapper, a tool from topological data analysis, reveals that fine-tuning restructures embedding space into modular, non-convex regions aligned with model predictions, even for highly ambiguous cases. Over $98\%$ of connected components exhibit $\geq 90\%$ prediction purity, yet alignment with ground-truth labels drops in ambiguous data, surfacing a hidden tension between structural confidence and label uncertainty. Unlike traditional tools such as PCA or UMAP, Mapper captures this geometry directly uncovering decision regions, boundary collapses, and overconfident clusters. Our findings position Mapper as a powerful diagnostic tool for understanding how models resolve ambiguity. Beyond visualization, it also enables topological metrics that may inform proactive modeling strategies in subjective NLP tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。