arXiv:2601.16863cs.AIcs.LG2026-01

用动态协作让多个小模型联手超越大模型性能。

Mixture-of-Models: Unifying Heterogeneous Agents via N-Way Self-Evaluating Deliberation

  • 运行时动态分配模型角色,像解背包问题一样优化选择。
  • 小模型组合在竞赛中媲美甚至超过百亿参数大模型。
  • 适合资源有限但追求高性能的开发者和研究者。

本文提出N-Way自评估协商(NSED)协议,一种运行时混合模型(MoM)架构,通过多个独立专家代理构建涌现式复合模型。不同于依赖静态门控网络的传统混合专家(MoE),NSED采用动态能力中介——一个将模型选择视为背包问题的运行时优化引擎,根据实时数据与成本约束为异构检查点分配功能角色。在执行层,将协商形式化为宏观循环神经网络(RNN),通过语义遗忘门实现迭代优化,且显存消耗不随迭代线性增长。关键组件包括无信任的N对N同行评审编排框架、非线性共识的二次投票激活函数,以及反馈驱动的状态更新。在AIME 2025和LiveCodeBench等挑战性基准上验证表明,由小于200亿参数的消费级模型组成的集合可匹敌或超越1000亿参数以上先进模型,确立了新的硬件性价比边界。此外,在DarkBench安全套件测试中显示内在对齐特性,同行协作纠正使谄媚评分低于任一单个代理。

原文摘要 · Abstract (English)

This paper introduces the N-Way Self-Evaluating Deliberation (NSED) protocol, a Runtime Mixture-of-Models (MoM) architecture that constructs emergent composite models from a plurality of distinct expert agents. Unlike traditional Mixture-of-Experts (MoE) which rely on static gating networks, NSED employs a Dynamic Expertise Broker - a runtime optimization engine that treats model selection as a variation of the Knapsack Problem, binding heterogeneous checkpoints to functional roles based on live telemetry and cost constraints. At the execution layer, we formalize deliberation as a Macro-Scale Recurrent Neural Network (RNN), where the consensus state loops back through a semantic forget gate to enable iterative refinement without proportional VRAM scaling. Key components include an orchestration fabric for trustless N-to-N peer review, a Quadratic Voting activation function for non-linear consensus, and a feedback-driven state update. Empirical validation on challenging benchmarks (AIME 2025, LiveCodeBench) demonstrates that this topology allows ensembles of small (less than 20B) consumer-grade models to match or exceed the performance of state-of-the-art 100B+ parameter models, establishing a new hardware arbitrage efficiency frontier. Furthermore, testing on the DarkBench safety suite reveals intrinsic alignment properties, with peer-mediated correction reducing sycophancy scores below that of any individual agent.

混合模型模型协作小模型大效能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。