用合成数据提升模型校准度，不降准确率
Fill In The Gaps: Model Calibration and Generalization with Synthetic Data
- 用大模型生成混合标签文本作为合成数据
- 测试显示准确率平均提升34%，校准误差降33%
- 适合需高可靠性的实际部署场景
随着机器学习模型快速发展，其性能校准成为实际应用前的关键挑战。现有校准方法因验证数据多样性不足，常损害模型准确率，降低泛化能力。为此，本文提出一种结合合成数据的校准方法，不牺牲准确率。基于概率近似正确（PAC）学习框架，推导出期望校准误差（ECE）上界。利用大语言模型（LLMs）生成具有混合类别标签的文本作为合成数据，有效降低ECE上界，并提升真实测试数据上的模型准确率。此外，设计了高效的合成数据生成机制。在四个自然语言处理任务上验证，平均准确率提升34%，ECE下降33%。
原文摘要 · Abstract (English)
As machine learning models continue to swiftly advance, calibrating their performance has become a major concern prior to practical and widespread implementation. Most existing calibration methods often negatively impact model accuracy due to the lack of diversity of validation data, resulting in reduced generalizability. To address this, we propose a calibration method that incorporates synthetic data without compromising accuracy. We derive the expected calibration error (ECE) bound using the Probably Approximately Correct (PAC) learning framework. Large language models (LLMs), known for their ability to mimic real data and generate text with mixed class labels, are utilized as a synthetic data generation strategy to lower the ECE bound and improve model accuracy on real test data. Additionally, we propose data generation mechanisms for efficient calibration. Testing our method on four different natural language processing tasks, we observed an average up to 34\% increase in accuracy and 33\% decrease in ECE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。