用大模型差异构建智能标签系统，精准处理联合国安理会决议文本。
Building an Ensemble LLM Semantic Tagger for UN Security Council Resolutions
- 通过多轮运行不同GPT模型，利用性能差异生成候选输出。
- GPT-4.1在清洗和标签任务中分别达到84.9%和99.99%的语义保留率。
- 仅用20%成本的小模型可接近最优效果，适合资源受限场景。
本文提出一种基于大模型的新型方法，用于对联合国安理会决议进行高精度、高效能的语义标注。核心思路是利用大模型间性能差异构建集成系统，以提升数据清洗与语义标注质量。为此引入两个评估指标：内容保留率（CPR）和标签规范性（TWF），防止幻觉及无关增删。实验表明，GPT-4.1在清洗任务中实现84.9%的CPR，语义标注任务中达到99.99%的CPR与99.92%的TWF。成本方面，较小的GPT-4.1-mini模型在两项任务中均达到接近最优表现，成本仅为前者20%。通过该集成设计与指标筛选，系统可从多轮运行中选取最佳输出，确保结果可靠，适用于复杂文本的自动化语义标注。
原文摘要 · Abstract (English)
This paper introduces a new methodology for using LLM-based systems for accurate and efficient semantic tagging of UN Security Council resolutions. The main goal is to leverage LLM performance variability to build ensemble systems for data cleaning and semantic tagging tasks. We introduce two evaluation metrics: Content Preservation Ratio (CPR) and Tag Well-Formedness (TWF), in order to avoid hallucinations and unnecessary additions or omissions to the input text beyond the task requirement. These metrics allow the selection of the best output from multiple runs of several GPT models. GPT-4.1 achieved the highest metrics for both tasks (Cleaning: CPR 84.9% - Semantic Tagging: CPR 99.99% and TWF 99.92%). In terms of cost, smaller models, such as GPT-4.1-mini, achieved comparable performance to the best model in each task at only 20% of the cost. These metrics ultimately allowed the ensemble to select the optimal output (both cleaned and tagged content) for all the LLM models involved, across multiple runs. With this ensemble design and the use of metrics, we create a reliable LLM system for performing semantic tagging on challenging texts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。