首篇系统梳理黑盒大模型校准方法的综述
A Survey of Calibration Process for Black-Box LLMs
- 将校准过程拆解为置信度估计与校准两个核心步骤
- 总结黑盒场景下可用的校准方法及实施挑战
- 适合关注大模型可靠性与人机对齐的研究者
大语言模型（LLMs）在语义理解与生成方面表现卓越，但其输出可靠性的准确评估仍是重大挑战。尽管已有大量研究探索校准技术，但多数聚焦于可访问参数的白盒模型。黑盒大模型虽性能更优，却因仅支持API交互，对校准技术提出更高要求。虽然近期研究已在黑盒校准上取得突破，但系统性综述仍属空白。为此，本文首次全面综述黑盒大模型的校准技术。我们首先将大模型校准过程定义为置信度估计与校准两个相互关联的关键步骤；其次，在黑盒设定下系统回顾适用方法，分析实现中的独特挑战与内在关联；进一步探讨该过程在黑盒大模型中的典型应用，并展望未来研究方向，为提升模型可靠性与人机对齐提供新视角。项目开源地址：https://github.com/LiangruXie/Calibration-Process-in-Black-Box-LLMs
原文摘要 · Abstract (English)
Large Language Models (LLMs) demonstrate remarkable performance in semantic understanding and generation, yet accurately assessing their output reliability remains a significant challenge. While numerous studies have explored calibration techniques, they primarily focus on White-Box LLMs with accessible parameters. Black-Box LLMs, despite their superior performance, pose heightened requirements for calibration techniques due to their API-only interaction constraints. Although recent researches have achieved breakthroughs in black-box LLMs calibration, a systematic survey of these methodologies is still lacking. To bridge this gap, we presents the first comprehensive survey on calibration techniques for black-box LLMs. We first define the Calibration Process of LLMs as comprising two interrelated key steps: Confidence Estimation and Calibration. Second, we conduct a systematic review of applicable methods within black-box settings, and provide insights on the unique challenges and connections in implementing these key steps. Furthermore, we explore typical applications of Calibration Process in black-box LLMs and outline promising future research directions, providing new perspectives for enhancing reliability and human-machine alignment. This is our GitHub link: https://github.com/LiangruXie/Calibration-Process-in-Black-Box-LLMs
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。