用分类器指导提示词,让大模型在信用违约预测中更准且召回更高。
When Does a Classifier Help an LLM? Classifier-Guided Prompting and Hybrid Classifier-LLM Models for Credit-Default Prediction
- 用分类器重要特征和预测概率构建提示词,提升LLM性能。
- 加了概率信息后,LLM的AUC-ROC达0.78,匹配随机森林。
- 该方法简单有效,适合金融领域需高召回的场景。
信用违约预测是金融决策中的关键任务。传统方法使用逻辑回归、随机森林等分类器对表格特征进行建模。近期大语言模型(LLMs)通过提示工程被应用于该任务。本文研究如何将训练好的分类器与LLM结合。区分直接让LLM模仿分类器与用分类器构建提示词两种方式。假设分类器可弥补LLM提示在排序能力上的不足。在Credit Card Clients数据集上实验,报告了召回率、F1值及ROC与精确率-召回率曲线下面积,并附自举置信区间。结果表明:少样本LLM具有最高召回率(0.47)和F1值(0.50),但排序能力较差(AUC-ROC 0.72 vs 随机森林0.79)。指令LLM模仿分类器无显著改善。仅保留分类器8个最重要特征构建提示词,召回率提升0.071,F1提升0.032。加入分类器预测概率后,LLM的AUC-ROC从0.72升至0.78,与随机森林相当,同时保持0.118更高的召回率。反向组合及多分类器使用无效。因此推荐采用简单的分类器引导提示词策略用于基于LLM的信用预测。
原文摘要 · Abstract (English)
Credit-default prediction is an important task in financial decision making. Traditional methods use fitted classifiers such as logistic regression and random forests on tabular features. Large language models (LLMs) have recently been applied to this task through prompting. In this work we study how a fitted classifier and an LLM can be combined for credit-default prediction. We distinguish telling the LLM to imitate a classifier from using the classifier to build the prompt. We hypothesize that a fitted classifier can supply the ranking ability that an LLM prompt lacks. We experiment on the Default of Credit Card Clients dataset, and report recall, F1, and the area under the ROC and precision-recall curves, with bootstrap confidence intervals. We observe that a few-shot LLM has the highest recall (0.47) and F1 (0.50) of any single model but ranks worse than a random forest (AUC-ROC 0.72 against 0.79). Instructing the LLM to imitate a classifier gives no significant change. Pruning the prompt to the classifier's eight most important features raises recall by 0.071 and F1 by 0.032. Adding the classifier's predicted probability to the prompt raises the LLM's AUC-ROC from 0.72 to 0.78, matching the random forest, while keeping 0.118 higher recall than it. The reverse composition, and the use of several classifiers, do not help. We thus recommend a simple classifier-guided prompt for LLM-based credit prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。