arXiv:2505.24622cs.AIcs.LG2025-05被引 1

用大模型生成简单问题,投票判断成功概率,透明可调控。

Random Rule Forest (RRF): Interpretable and Manageable Ensembles of LLM-Generated Questions for Predicting Success from Unstructured Data

  • 用LLM生成一系列简单是/否问题,作为弱学习器
  • 在初创企业筛选中精度达基线数倍,临床试验任务上优于已有基线
  • 适合对可解释性要求高的高风险决策场景

许多高风险筛选任务需从非结构化文本中预测罕见结果,错误代价高昂且决策必须可审计。我们提出随机规则森林(RRF),一种可解释的集成方法,不将大语言模型(LLM)作为端到端预测器,而是用其生成简单的是/否问题。每个问题作为弱学习器,其回答通过简单等权重投票形成可审计的“绿灯”评分卡:足够多独立的正向信号表明成功概率更高。我们认为,在正样本稀少、难以估计学习权重时,这种刻意的简洁性是一种稳健的默认选择。我们在两个低发生率领域评估了RRF。在早期创业公司筛选中,基于创始人资料,RRF生成透明评分卡,其精确度为基线数倍(少量专家输入可进一步提升),且操作点可直接控制,优于直接提示;在已建立的I期临床试验基准上,RRF在阈值无关指标PR-AUC和ROC-AUC上优于已有基线。这些结果表明,LLM可作为高风险文本决策中的可审计特征生成器,兼具透明性与竞争力的预测性能。

原文摘要 · Abstract (English)

Many high-stakes screening tasks require predicting rare outcomes from unstructured text, where errors are costly and decisions must be auditable. We introduce Random Rule Forest (RRF), an interpretable ensemble that uses a large language model (LLM) not as an end-to-end predictor but as a generator of simple YES/NO questions. Each question acts as a weak learner, and their responses are combined by a plain unit-weight vote into an auditable ``green-flags'' scorecard: enough independent positive signals indicate a higher chance of success. We argue this deliberate simplicity is a robust default when positives are scarce and learned weights are hard to estimate. We evaluate RRF in two low-base-rate domains. On early-stage startup screening from founder profiles, RRF produces a transparent scorecard whose precision is several times the base rate (with light expert input raising it further) and, unlike direct prompting, its operating point can be controlled directly. On an established Phase~I clinical-trial benchmark, RRF outperforms published baselines on the threshold-independent metrics PR-AUC and ROC-AUC. Together these show that LLMs can serve as auditable feature generators for high-stakes text-based decisions, combining transparency with competitive predictive performance.

可解释性大模型应用低频预测评分卡

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。