用统计与智能体推理结合，用少量数据精准预测大模型性能。
STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction
- 融合外部知识与概率矩阵分解，生成带不确定性的性能预期。
- 在每模型仅1-2个观测值时，性能预测总分提升14.46%。
- 适合需要高效评估大模型的团队，解释可追溯。
随着大规模模型全面评估成本高昂，从有限观测中预测模型性能变得至关重要。现有统计方法难以应对模式漂移、数据稀疏且缺乏可解释性，而纯大模型方法又不可靠。我们提出STAR框架，将数据驱动的统计期望与知识驱动的智能体推理相结合。STAR利用专用检索器获取外部知识，并将语义特征嵌入受限概率矩阵分解（CPMF）中，生成带有不确定性的统计期望。一个基于期望偏离理论（EVT）引导的推理模块，通过同家族分析、跨模型比较和可信度感知聚合，对预测进行优化，生成可追溯的修正结果。大量实验表明，STAR在评分与排序指标上均显著优于所有基线，在极端稀疏条件下（每测试模型仅1–2个观测分数），相比最强统计方法总分提升14.46%。
原文摘要 · Abstract (English)
As comprehensive large model evaluation becomes prohibitively expensive, predicting model performance from limited observations has become essential. However, existing statistical methods struggle with pattern shifts, data sparsity, and lack of explanation, while pure LLM methods remain unreliable. We propose STAR, a framework that bridges data-driven STatistical expectations with knowledge-driven Agentic Reasoning. STAR leverages specialized retrievers to gather external knowledge and embeds semantic features into Constrained Probabilistic Matrix Factorization (CPMF) to generate statistical expectations with uncertainty. A reasoning module guided by Expectation Violation Theory (EVT) then refines predictions through intra-family analysis, cross-model comparison, and credibility-aware aggregation, producing adjustments with traceable explanations. Extensive experiments show that STAR consistently outperforms all baselines on both score-based and rank-based metrics, delivering a 14.46% gain in total score over the strongest statistical method under extreme sparsity, with only 1--2 observed scores per test model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。