用遗传编程生成可解释的分类器，让Transformer嵌入变公式
From Embeddings to Equations: Genetic-Programming Surrogates for Interpretable Transformer Classification
- 通过语义保真分割嵌入，用遗传算法学出可读公式
- 在多个数据集上达到近0.99的F1，且校准误差显著降低
- 适合需要可解释性与高可信度的场景
本文研究冻结Transformer嵌入的符号代理建模，以获得紧凑、可审计且概率校准的分类器。针对五个基准数据集（SST2G、20NG、MNIST、CIFAR10、MSC17），使用ModernBERT、DINOv2和SigLIP提取的嵌入，通过语义保真特征分割（SPFP）划分为不相交但信息保留的视图。随后，采用合作多族群遗传程序（MEGP）在这些视图上学习加性闭式对数几率程序。每数据集进行30次运行，报告F1、AUC、log-loss、Brier、期望校准误差（ECE）及符号复杂度；基于验证集F1的1标准误差规则选取典型模型，并以简约性为优先。在验证集上进行温度缩放后，测试集的ECE显著下降。所得代理模型在多数数据集上表现优异（如在MNIST、CIFAR10、MSC17上F1达约0.99，在SST2G上约0.95），而20NG仍具挑战性。提供可靠性图、维度使用与重叠统计、贡献重要性及全局效应分析（PDP与ALE），验证其解释力跨模态且基于显式程序。
原文摘要 · Abstract (English)
We study symbolic surrogate modeling of frozen Transformer embeddings to obtain compact, auditable classifiers with calibrated probabilities. For five benchmarks (SST2G, 20NG, MNIST, CIFAR10, MSC17), embeddings from ModernBERT, DINOv2, and SigLIP are partitioned on the training set into disjoint, information-preserving views via semantic-preserving feature partitioning (SPFP). A cooperative multi-population genetic program (MEGP) then learns additive, closed-form logit programs over these views. Across 30 runs per dataset we report F1, AUC, log-loss, Brier, expected calibration error (ECE), and symbolic complexity; a canonical model is chosen by a one-standard-error rule on validation F1 with a parsimony tie-break. Temperature scaling fitted on validation yields substantial ECE reductions on test. The resulting surrogates achieve strong discrimination (up to F1 around 0.99 on MNIST, CIFAR10, MSC17; around 0.95 on SST2G), while 20NG remains most challenging. We provide reliability diagrams, dimension usage and overlap statistics, contribution-based importances, and global effect profiles (PDP and ALE), demonstrating faithful, cross-modal explanations grounded in explicit programs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。