提出公平评估测试时计算的协议，解决模型性能波动带来的评估失真问题。
FEval-TTC: Fair Evaluation Protocol for Test-Time Compute
- 统一多模型、多数据集的推理评估流程，标准化提示与答案提取。
- 支持数学与常识推理任务，可量化每查询的令牌与美元成本。
- 开源评估框架，适合研究测试时计算方法的学者使用。
大型语言模型（LLMs）的性能和API调用成本会随时间波动，可能使先前研究结论失效。为此，我们提出公平评估测试时计算（FEval-TTC）协议，确保无论性能如何波动，对测试时计算（TTC）方法的评估均保持一致。该协议聚焦于基于思维链（CoT）的TTC方法，支持在多种数学与常识推理数据集上跨多个大模型进行评估。通过标准化少样本提示与答案提取流程，显著降低研究人员的时间与金钱开销。此外，提供成本建模方法，可估算每查询的令牌消耗与美元成本，实现主流TTC方法间的公平比较。相关代码已开源，地址为 https://github.com/networkslab/feval_ttc。
原文摘要 · Abstract (English)
The performance of Large Language Models (LLMs) and the associated dollar costs of API calls can fluctuate over time, potentially invalidating conclusions drawn in prior research. To address this, we propose a Fair Evaluation protocol for Test-Time Compute (FEval-TTC), designed to ensure consistent assessment of test-time compute (TTC) methods, regardless of such fluctuations. FEval-TTC focuses on the evaluation of TTC methods that utilize underlying Chains-of-Thought (CoT). It supports evaluations across multiple LLMs on a diverse set of mathematical and commonsense reasoning datasets. The few-shot prompting and answer extraction processes are standardized across datasets, reducing both time and monetary overhead for researchers. Furthermore, we provide a cost modelling procedure that estimates both the token and dollar cost per query, facilitating equitable comparisons of prevalent TTC methods. We open-source FEval-TTC for public use at https://github.com/networkslab/feval_ttc .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。