arXiv:2508.04219cs.CLcs.LG2025-08被引 2

用大模型API做文本分级分类,选对提示策略能提升效果

Hierarchical Text Classification Using Black Box Large Language Models

  • 通过三种提示策略利用黑盒大模型进行分级分类
  • 少样本下准确率优于传统方法,深层层级表现更优
  • 提示策略影响成本与效果,需权衡使用

层次化文本分类(HTC)旨在将文本分配到结构化的标签层级中,但受限于数据稀缺和模型复杂性。本研究探索通过API调用黑盒大语言模型(LLM)实现HTC的可行性,作为替代传统机器学习方法的方案,后者需要大量标注数据和计算资源。我们评估了三种提示策略:直接叶节点预测(DL)、直接层级预测(DH)和自顶向下多步层级预测(TMH),在零样本和少样本设置下比较其准确率与成本效益。在两个数据集上的实验表明,少样本设置下的分类准确率始终高于零样本设置。尽管传统机器学习模型在浅层级数据集上表现优异,但在深层层级数据集上,尤其是采用DH策略的LLM表现更优。然而,由于深度层级所需的输入标记数更多,DH策略的API成本显著上升。结果凸显了准确率提升与提示策略计算成本之间的权衡。这些发现表明黑盒LLM在HTC中的潜力,同时强调需谨慎选择提示策略以平衡性能与成本。

原文摘要 · Abstract (English)

Hierarchical Text Classification (HTC) aims to assign texts to structured label hierarchies; however, it faces challenges due to data scarcity and model complexity. This study explores the feasibility of using black box Large Language Models (LLMs) accessed via APIs for HTC, as an alternative to traditional machine learning methods that require extensive labeled data and computational resources. We evaluate three prompting strategies -- Direct Leaf Label Prediction (DL), Direct Hierarchical Label Prediction (DH), and Top-down Multi-step Hierarchical Label Prediction (TMH) -- in both zero-shot and few-shot settings, comparing the accuracy and cost-effectiveness of these strategies. Experiments on two datasets show that a few-shot setting consistently improves classification accuracy compared to a zero-shot setting. While a traditional machine learning model achieves high accuracy on a dataset with a shallow hierarchy, LLMs, especially DH strategy, tend to outperform the machine learning model on a dataset with a deeper hierarchy. API costs increase significantly due to the higher input tokens required for deeper label hierarchies on DH strategy. These results emphasize the trade-off between accuracy improvement and the computational cost of prompt strategy. These findings highlight the potential of black box LLMs for HTC while underscoring the need to carefully select a prompt strategy to balance performance and cost.

文本分类大模型应用提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。