用教师模型概率分布指导学生，提升Tsetlin机性能不增加延迟
A Novel Approach To Implementing Knowledge Distillation In Tsetlin Machines
- 用教师输出概率分布为学生提供额外上下文
- 学生模型准确率显著提升，延迟无明显增加
- 适合需要高效推理的图像与文本分类场景
Tsetlin机器是一种基于命题逻辑的模型，通过合取子句从数据中学习模式。与典型神经网络类似，其性能高度依赖参数数量：参数越多，准确率越高，但执行速度越慢。在神经网络中，知识蒸馏可将训练好的教师模型信息迁移到更小的学生模型，以在不增加推理时间的前提下提升学生模型的准确率。本文提出一种新颖的知识蒸馏方法，利用教师模型每个输出样本的概率分布，为学生模型提供额外上下文信息。此外，提出一种新的条款迁移算法，根据教师模型中各条款的重要性进行加权，并仅将最核心的条款初始化到学生模型中。实验表明，该方法在图像识别和文本分类任务中能显著提升学生模型性能，且未对延迟产生负面影响。
原文摘要 · Abstract (English)
The Tsetlin Machine (TM) is a propositional logic based model that uses conjunctive clauses to learn patterns from data. As with typical neural networks, the performance of a Tsetlin Machine is largely dependent on its parameter count, with a larger number of parameters producing higher accuracy but slower execution. Knowledge distillation in neural networks transfers information from an already-trained teacher model to a smaller student model to increase accuracy in the student without increasing execution time. We propose a novel approach to implementing knowledge distillation in Tsetlin Machines by utilizing the probability distributions of each output sample in the teacher to provide additional context to the student. Additionally, we propose a novel clause-transfer algorithm that weighs the importance of each clause in the teacher and initializes the student with only the most essential data. We find that our algorithm can significantly improve performance in the student model without negatively impacting latency in the tested domains of image recognition and text classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。