为去中心化大模型推理设计多维质量评分框架,提升输出可信度。
A Multi-Dimensional Quality Scoring Framework for Decentralized LLM Inference with Proof of Quality
- 将输出质量拆解为结构、语义、对齐等六大维度,模块化评估
- 实测发现部分维度与真实质量负相关,需校准权重后才有效
- 可直接接入PoQ系统,抗攻击能力强,适合分布式推理场景
去中心化大语言模型推理网络可聚合异构算力以扩展服务规模,但需轻量且激励相容的机制来评估输出质量。现有工作提出成本感知的Proof of Quality(PoQ)和自适应鲁棒PoQ,以应对评估者异质性和恶意行为。本文聚焦质量信号本身,提出一种多维质量评分框架,将输出质量分解为模型与成本先验、结构质量、语义质量、查询-输出对齐度以及一致性/不确定性六大模块。基于问答与摘要任务的记录输出,系统审计各维度可靠性,发现看似合理的维度在不同任务中表现不一,甚至与参考质量呈负相关,需校准。默认复合评分表现不佳,但通过剔除不可靠维度并重新归一化权重,得到校准后的复合评分,其性能达到或超过最优单一分评器及共识基线。最终,将该复合评分作为即插即用的质量信号集成至PoQ,在对抗性评估者攻击下,结合鲁棒聚合与自适应信任加权,展现出互补优势。
原文摘要 · Abstract (English)
Decentralized large language model (LLM) inference networks can pool heterogeneous compute to scale serving, but they require lightweight and incentive-compatible mechanisms to assess output quality. Prior work introduced cost-aware Proof of Quality (PoQ) and adaptive robust PoQ to allocate rewards under evaluator heterogeneity and adversarial behavior. In this paper, we focus on the quality signal itself and propose a multi-dimensional quality scoring framework that decomposes output quality into modular dimensions, including model and cost priors, structure quality, semantic quality, query-output alignment, and agreement/uncertainty. Using logged outputs from QA and summarization tasks, we systematically audit dimension reliability and show that seemingly reasonable dimensions can be task-dependent and even negatively correlated with reference quality without calibration. While the default composite underperforms a strong single semantic evaluator, ablations reveal that removing unreliable dimensions and re-normalizing weights yields a calibrated composite that matches or exceeds the best single- evaluator and consensus baselines. Finally, we integrate the composite score as a drop-in quality signal in PoQ and demonstrate complementary benefits with robust aggregation and adaptive trust weighting under adversarial evaluator attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。