提出新层次聚类方法,能自动判断是否应存在层级结构。
Hierarchical Linkage Clustering Beyond Binary Trees and Ultrametrics
- 用偏序关系定义有效层次结构,突破二叉树限制
- 可生成最细粒度有效层级,无层级时退化为星形结构
- 适用于任意满足条件的链接函数,经典方法均适用
层次聚类旨在通过构建簇的树状结构揭示数据中的嵌套关系,但传统方法存在三大缺陷:(i) 总返回一个层次结构,即使数据中并不存在;(ii) 强制要求二叉树,无法表达非二叉的真实结构;(iii) 对链接函数选择高度敏感。本文引入‘有效层次’概念,并在所有有效层次上定义偏序关系,证明了最精细有效层次的存在性——即在保持数据相似性结构的前提下编码最多信息的层次。该层次不限于二叉结构,当无真实层级关系时会退化为星形树。我们提出两步算法:先用链接法构造二叉树,再通过剪枝强制有效性。建立了链接函数的充要条件,使得该过程可精确恢复最精细有效层次,且所有满足条件的链接函数经剪枝后结果一致。经典方法如单链接、全链接、平均链接均满足条件,而Ward链接不满足。
原文摘要 · Abstract (English)
Hierarchical clustering seeks to uncover nested structures in data by constructing a tree of clusters, where deeper levels reveal finer-grained relationships. Traditional methods, including linkage approaches, face three major limitations: (i) they always return a hierarchy, even if none exists, (ii) they are restricted to binary trees, even if the true hierarchy is non-binary, and (iii) they are highly sensitive to the choice of linkage function. In this paper, we address these issues by introducing the notion of a valid hierarchy and defining a partial order over the set of valid hierarchies. We prove the existence of a finest valid hierarchy, that is, the hierarchy that encodes the maximum information consistent with the similarity structure of the data set. In particular, the finest valid hierarchy is not constrained to binary structures and, when no hierarchical relationships exist, collapses to a star tree. We propose a simple two-step algorithm that first constructs a binary tree via a linkage method and then prunes it to enforce validity. We establish necessary and sufficient conditions on the linkage function under which this procedure exactly recovers the finest valid hierarchy, and we show that all linkage functions satisfying these conditions yield the same hierarchy after pruning. Notably, classical linkage rules such as single, complete, and average satisfy these conditions, whereas Ward's linkage fails to do so.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。