提出轻量级无参考质量评估框架，用于去中心化大模型推理的成本感知验证。
PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference
- 训练专用判别模型，无需真实答案即可评估输出质量。
- 最高达0.747的皮尔逊相关性，优于已有基于参考的答案评估方法。
- 支持在线校准与级联评估，成本降低72.7%且质量损失小，适合部署优化。
去中心化大模型推理网络需要轻量级、无参考的质量评估来实现证明质量（PoQ）。我们提出PoQ-Judge框架，通过训练专用判别模型对查询-输出对进行评分，无需真实参考答案。研究了三种架构在质量-成本权衡下的表现：TextCNN判别器、MiniLM交叉编码器和DeBERTa判别器。在UltraFeedback数据集与GPT标注的领域内数据上进行两阶段训练后，最佳模型在保留测试集上与真实代理指标达到0.747的皮尔逊相关性，超越以往基于参考的方法。作为复合评分中的无参考组件，其相关性达0.645，媲美最优单个基于参考的评估器，同时无需参考答案。我们还发现在线校准可识别语义质量为主要维度，级联评估使成本降低72.7%，质量损失轻微。结果在问答任务中显著优于摘要任务，表明代理质量仍是主要瓶颈。
原文摘要 · Abstract (English)
Decentralized LLM inference networks need lightweight, reference-free quality evaluation for Proof of Quality (PoQ). We present PoQ-Judge, a framework that trains dedicated judge models to score query-output pairs without ground-truth references. We study three architectures across the quality-cost tradeoff: a TextCNN judge, a MiniLM cross-encoder, and a DeBERTa judge. Using two-stage training on UltraFeedback plus GPT-labeled in-domain data, the best model reaches 0.747 Pearson correlation with the ground-truth proxy on a held-out test set, outperforming reference-based evaluators from prior work. As a reference-free component in composite scoring, it achieves 0.645 Pearson correlation, matching the best single reference-based evaluator while removing the need for reference answers. We also show that online calibration identifies semantic quality as the dominant dimension and that cascade evaluation reduces cost by 72.7 percent with only modest quality loss. Results are much stronger on QA than summarization, pointing to proxy quality as the main remaining limitation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。