对比10个主流大模型在文本分类中的表现，发现规模不是关键，协作才更有效。
Order from Chaos: Comparative Study of Ten Leading LLMs on Unstructured Data Categorization
- 统一数据集与零样本提示，公平比较十款大模型分类能力。
- 平均准确率仅34%，多数模型存在过度分类和幻觉问题。
- 多模型协同策略显著提升效果，消除幻觉且性能超越单一模型。
本研究对十种先进大语言模型（LLMs）在使用互动广告局（IAB）2.2层级分类体系进行非结构化文本分类任务中的表现进行了对比评估。分析采用8,660条人工标注样本的统一数据集，并设置相同的零样本提示以保证方法一致性。评估指标包括准确率、精确率、召回率、F1分数四项经典指标，以及幻觉率、膨胀率和分类成本三项大模型特有指标。结果显示，尽管模型快速演进，当前主流模型的经典性能仅为中等水平，平均准确率为34%，精确率为42%，召回率为45%，F1分数为41%。幻觉率与膨胀率表明，模型频繁生成超出人类标注者范围的类别。在评估系统中，Gemini 1.5/2.0 Flash 和 GPT 20B/120B 在性价比上表现最优，而 GPT 120B 幻觉率最低。研究指出，单纯扩大规模或改进架构不足以提升分类准确性，因该任务需将丰富非结构化文本压缩至有限分类体系，对现有模型架构构成挑战。为此，本文提出并测试了一种基于集成的方法：多个大模型作为独立专家协同工作，该方法显著提升准确率，降低膨胀率，并完全消除幻觉。结果表明，模型间的协调调度，而非单纯规模扩张，可能是实现甚至超越人类专家水平大规模文本分类的有效路径。
原文摘要 · Abstract (English)
This study presents a comparative evaluation of ten state-of-the-art large language models (LLMs) applied to unstructured text categorization using the Interactive Advertising Bureau (IAB) 2.2 hierarchical taxonomy. The analysis employed a uniform dataset of 8,660 human-annotated samples and identical zero-shot prompts to ensure methodological consistency across all models. Evaluation metrics included four classic measures - accuracy, precision, recall, and F1-score - and three LLM-specific indicators: hallucination ratio, inflation ratio, and categorization cost. Results show that, despite their rapid advancement, contemporary LLMs achieve only moderate classic performance, with average scores of 34% accuracy, 42% precision, 45% recall, and 41% F1-score. Hallucination and inflation ratios reveal that models frequently overproduce categories relative to human annotators. Among the evaluated systems, Gemini 1.5/2.0 Flash and GPT 20B/120B offered the most favorable cost-to-performance balance, while GPT 120B demonstrated the lowest hallucination ratio. The findings suggest that scaling and architectural improvements alone do not ensure better categorization accuracy, as the task requires compressing rich unstructured text into a limited taxonomy - a process that challenges current model architectures. To address these limitations, a separate ensemble-based approach was developed and tested. The ensemble method, in which multiple LLMs act as independent experts, substantially improved accuracy, reduced inflation, and completely eliminated hallucinations. These results indicate that coordinated orchestration of models - rather than sheer scale - may represent the most effective path toward achieving or surpassing human-expert performance in large-scale text categorization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。