用大模型零样本生成可解释决策树，性能优于传统小数据方法
"Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Models
- 不依赖训练数据，直接用大模型知识生成决策树
- 在小数据集上表现超越数据驱动的决策树
- 适合低数据场景下的可解释建模与知识利用
大型语言模型（LLM）在数据稀缺时可借助其压缩的世界知识进行预测建模。本文展示如何利用LLM生成无需训练数据的内在可解释机器学习模型——决策树。我们发现，这些零样本决策树在某些小型表格数据集上甚至优于数据驱动的决策树；且由此生成的树嵌入在平均表现上也优于数据驱动的树基嵌入。因此，该方法可作为低数据环境下数据驱动方法的新知识驱动基线，并为利用LLM中丰富的世界知识解决表格机器学习任务提供新途径。代码与结果见https://github.com/ml-lab-htw/llm-trees。
原文摘要 · Abstract (English)
Large language models (LLMs) provide powerful means to leverage prior knowledge for predictive modeling when data is limited. In this work, we demonstrate how LLMs can use their compressed world knowledge to generate intrinsically interpretable machine learning models, i.e., decision trees, without any training data. We find that these zero-shot decision trees can even surpass data-driven trees on some small-sized tabular datasets and that embeddings derived from these trees perform better than data-driven tree-based embeddings on average. Our decision tree induction and embedding approaches can therefore serve as new knowledge-driven baselines for data-driven machine learning methods in the low-data regime. Furthermore, they offer ways to harness the rich world knowledge within LLMs for tabular machine learning tasks. Our code and results are available at https://github.com/ml-lab-htw/llm-trees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。