用语言模型聚类语义，让可解释的图灵机媲美BERT
Clusters are All You Need: Pre-Training the Tsetlin Machine with Semantic Clusters from Language Models for Interpretability
- 用K-means或Top2Vec对文本聚类，生成语义一致的样本组
- 通过增强型反馈机制训练非否定图灵机，准确率超基线模型
- 兼顾可解释性与性能，适合医疗、金融等高风险决策场景
预训练语言模型如BERT在文本分类上表现优异但缺乏透明性，限制了其在高风险场景的应用。图灵机（TM）提供完全可解释的基于子句的推理，但难以捕捉语义信息。此前方法依赖静态词嵌入，忽略上下文语义。本文提出一种语义预训练框架，将预训练语言模型的知识迁移至图灵机，无需使用词嵌入。通过K-means或Top2Vec将文本样本聚类为语义一致的簇，利用簇-样本对以增强型类型I反馈预训练非否定图灵机。该方法使图灵机学习可解释的语义关键词，并在下游任务中进行微调。在五个数据集上，该方法显著优于原始及基于嵌入的图灵机，性能接近BERT，同时保持高度可解释性。
原文摘要 · Abstract (English)
Pre-trained language models such as BERT achieve strong text classification performance but lack transparency, limiting their use in high-stakes settings. The Tsetlin Machine (TM) offers fully interpretable, clause-based reasoning but captures little semantic information, and prior attempts to bridge the two rely on static word embeddings that miss contextual meaning. We propose a semantic pre-training framework that transfers knowledge from a pre-trained language model into a TM without using embeddings. Text samples are grouped into semantically coherent clusters with K-means or Top2Vec, and the resulting cluster-sample pairs pre-train a non-negated TM with enhanced Type I feedback. The TM thereby learns interpretable semantic keywords that are fine-tuned on downstream tasks. Across five datasets, our method substantially outperforms vanilla and embedding-based TMs and reaches performance competitive with BERT while remaining interpretable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。