用大模型生成信贷风险解释,发现效果差且不可靠。
Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk

- 构建多尺度堆叠集成模型,信用评分ROC-AUC达0.9539
- 大模型生成的解释存在严重错误,关键因素识别率低
- 需验证生成内容真实性,仅靠提示工程无法解决
信贷评分日益依赖决策逻辑不可读的模型,与监管要求相冲突。一种常见方案是用语言模型生成解释:先计算特征重要性,再交由LLM撰写理由。我们端到端构建该系统并检验其有效性。预测部分采用多尺度堆叠集成,融合四种正则化梯度提升学习器与残差网络,通过神经元元学习器训练,基于32,581条公开申请数据,测试集ROC-AUC为0.9539(95%置信区间[0.9462, 0.9616]),PR-AUC为0.9137,优于最优单模型0.0143(p=0.016)。核心发现呈不对称性:排名提升真实但实际影响微小,在F1最优阈值下仅比调优随机森林多避免6个漏判违约,成本加权损失下降不足2%。叙事层失效,仅靠提示工程无法修复:审计案例中,模型将三个实际降低风险的因素误标为风险增加,遗漏主导因子,并引入未提供特征。根源在于:SHAP与LIME虽在重要性排序上一致(重合度@10=0.80),但在顺序上差异显著(tau=0.43, p=0.18),且模型最敏感输入的归因符号在申请人中近乎随机(模态符号占比0.53)。校准(ECS=0.117)和扰动稳定性(DPD=0.078)均未达标。约束提示必要但不充分,生成后必须验证,不能假设。
原文摘要 · Abstract (English)
Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters, in tension with supervisory expectations that adverse decisions be explainable. A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale. We build such a system end to end and test whether the second half of the promise holds. The predictive component is a multi-scale stacking ensemble fusing four differently regularised gradient-boosting learners with a residual network through a neural meta-learner trained on out-of-fold predictions. On a public 32,581-application credit dataset it reaches test ROC-AUC 0.9539 (95% CI [0.9462, 0.9616]) and PR-AUC 0.9137, beating the best single model by Delta-AUC = 0.0143 (p = 0.016 under a conservative independence assumption). Our central finding is asymmetric. The ranking gain is real but operationally small: at the F1-optimal threshold the ensemble avoids only six additional missed defaults out of 1,422 against a tuned random forest, cutting cost-weighted loss by under 2%. The narrative layer fails in a way prompt engineering alone does not fix. In an audited case the model named three factors as risk-increasing that the supplied attributions scored as risk-reducing, omitted the dominant driver, and introduced a feature never given to it. We trace this to properties we measure rather than assume: SHAP and LIME agree on which features matter (overlap@10 = 0.80) but not on their order (tau = 0.43, p = 0.18), and the attribution sign for the model's most sensitive input is near a coin flip across applicants (modal-sign share 0.53). Calibration (ECS = 0.117) and perturbation stability (DPD = 0.078) both fall short of our own thresholds. Constrained prompting is necessary but not sufficient: grounding must be verified after generation, not assumed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。