用100个精选样本高效评估大模型可靠性，节省90%成本。
MicroProbe: Efficient Reliability Assessment for Foundation Models with Minimal Data
- 通过五维提示多样性与自适应加权，精准定位潜在失效模式。
- 相比随机采样，综合可靠性得分提升23.5%，统计显著（p<0.001）。
- 适合需快速验证AI安全性的研发与部署团队使用。
大模型可靠性评估通常需要数千个测试样本，计算成本高且耗时长。本文提出MicroProbe，仅需100个精心挑选的探针样本即可完成全面可靠性评估。该方法结合五个关键可靠性维度的策略性提示多样性、先进的不确定性量化与自适应加权机制，高效识别潜在失败模式。在多个语言模型（GPT-2变体、GPT-2 Medium、GPT-2 Large）及跨领域验证（医疗、金融、法律）中，MicroProbe相比随机采样基线实现23.5%更高的综合可靠性评分，具有极强统计显著性（p < 0.001，Cohen's d = 1.21）。三位人工智能安全专家评估确认，本方法战略选择有效性得分为4.14/5.0，显著高于随机采样的3.14/5.0。MicroProbe以99.9%统计功效完成评估，评估成本降低90%，同时保持传统方法95%的覆盖率。该方法填补了负责任AI部署中高效模型评估的关键空白。
原文摘要 · Abstract (English)
Foundation model reliability assessment typically requires thousands of evaluation examples, making it computationally expensive and time-consuming for real-world deployment. We introduce microprobe, a novel approach that achieves comprehensive reliability assessment using only 100 strategically selected probe examples. Our method combines strategic prompt diversity across five key reliability dimensions with advanced uncertainty quantification and adaptive weighting to efficiently detect potential failure modes. Through extensive empirical evaluation on multiple language models (GPT-2 variants, GPT-2 Medium, GPT-2 Large) and cross-domain validation (healthcare, finance, legal), we demonstrate that microprobe achieves 23.5% higher composite reliability scores compared to random sampling baselines, with exceptional statistical significance (p < 0.001, Cohen's d = 1.21). Expert validation by three AI safety researchers confirms the effectiveness of our strategic selection, rating our approach 4.14/5.0 versus 3.14/5.0 for random selection. microprobe completes reliability assessment with 99.9% statistical power while representing a 90% reduction in assessment cost and maintaining 95% of traditional method coverage. Our approach addresses a critical gap in efficient model evaluation for responsible AI deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。