无需参考答案，自动融合多个大模型判断结果
SkillAggregation: Reference-free LLM-Dependent Aggregation
- 基于大模型自身输出构建可学习的融合机制
- 在无参考数据下性能超越传统方法，多任务表现最优
- 适合没有标注数据、需集成多个模型判断的场景
大语言模型因其生成类人判断的能力，正被广泛用于评估自然语言处理任务。早期仅使用单一模型，但近期研究发现多个模型联合判断能提升效果。关键在于如何聚合这些判断。现有方法或对所有模型赋相同权重，或仅适用于特定任务（如幻觉检测）。本文提出一种名为SkillAggregation的新方法，可在无参考标签的情况下，从多个模型中聚合预测结果，且无需额外数据或真实标签。该方法扩展了原本用于图像分类的Crowdlayer聚合框架，利用推理阶段的模型判断进行自适应融合。在HaluEval-Dialogue、TruthfulQA和Chatbot Arena三个任务上与多种标准聚合方法对比，SkillAggregation在所有任务上均优于Crowdlayer，多数任务中表现最佳。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly used to assess NLP tasks due to their ability to generate human-like judgments. Single LLMs were used initially, however, recent work suggests using multiple LLMs as judges yields improved performance. An important step in exploiting multiple judgements is the combination stage, aggregation. Existing methods in NLP either assign equal weight to all LLM judgments or are designed for specific tasks such as hallucination detection. This work focuses on aggregating predictions from multiple systems where no reference labels are available. A new method called SkillAggregation is proposed, which learns to combine estimates from LLM judges without needing additional data or ground truth. It extends the Crowdlayer aggregation method, developed for image classification, to exploit the judge estimates during inference. The approach is compared to a range of standard aggregation methods on HaluEval-Dialogue, TruthfulQA and Chatbot Arena tasks. SkillAggregation outperforms Crowdlayer on all tasks, and yields the best performance over all approaches on the majority of tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。