arXiv:2607.04668cs.OScs.AI2026-07

让大模型推理动态调整核心数,实现毫秒级无锁协同。

Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes

  • 通过确认制时钟协议实现每令牌动态换人,不阻塞也不丢数据。
  • 在8核机器上实测7B模型输出完全一致,推理吞吐最高提升1.75倍。
  • 适合需高并发的嵌入式设备,兼顾推理与系统任务调度需求。

设备端大模型解码是毫秒级硬屏障的CPU-SIMD计算,要求每个令牌独占全部核心,但操作系统又需要持续使用这些核心。传统抢占式调度无法直接兼容:未通知离开会死锁,未通知到达会破坏逻辑值。本文提出Anima OS中的弹性推理组(elastic gang),一个裸金属x86-64 Rust内核,将推理组作为可调度实体,允许任意两令牌间变更核心成员。核心机制为带确认的时钟周期协议,不依赖特定核心;每个令牌参与者集合是请求核心与已确认当前周期核心的交集。未确认核心不在当前令牌中,最多延迟一令牌加入。被驱逐的一般进程迁移并继续运行,核心在周期结束即返回。在真实AMD Zen 5机器(8核/16线程)上,对135M和7B模型验证了每令牌成员变更下的比特精确输出,确保内核安全门读取逻辑值时的可靠性。相比公平静态核心划分,在中等推理负载下,弹性成员配置以同等或更优的推理吞吐,分别实现1.75倍(25%)、1.52倍(50%)、1.28倍(75%)的通用任务吞吐,推理空闲时可回收全部8个核心,并在饱和时收敛至静态划分。归还借出核心耗时0.22微秒(中位数),获取已被租用的核心需一个调度周期(约16毫秒),租户从不被中途抢占。解码吞吐在8核时饱和,超过拐点后出让核心几乎无代价,弹性自动在线调节推理组规模。

原文摘要 · Abstract (English)

On-device LLM decoding is a hard-barriered CPU-SIMD computation that wants every core for milliseconds per token, while the rest of the OS wants those same cores continuously. A barriered gang cannot simply be dropped into a preemptive scheduler: an unannounced departure deadlocks a barrier, and an unannounced arrival silently corrupts logits. I present the elastic gang of Anima OS, a bare-metal x86-64 Rust kernel in which the inference gang is a first-class schedulable entity whose core membership may change between any two tokens. The core mechanism is an ACK-latched epoch protocol that never waits on a named core: a seqlock-style generation-tagged latch composed with RCU/epoch-style membership consent, so each token's participant set is the intersection of the cores the gang requested and the cores that acked the current epoch. An un-acked core is outside this token and joins at most one token later. Displaced general processes migrate and keep running; cores return to them the moment a generation ends. On a real AMD Zen 5 machine (8C/16T), inference output is bit-exact under verified per-token membership change on both a 135M and a 7B model, the property that makes elasticity safe in a kernel whose safety gate reads logits. Against fair static core partitions, elastic membership Pareto-dominates: at intermediate inference duty cycles it delivers 1.75x (25%), 1.52x (50%), and 1.28x (75%) the general throughput of a static 8-core split at equal or better inference throughput, recovers all eight stranded cores when inference is idle, and converges to the split at saturation. Returning a lent core costs 0.22 us (p50); acquiring a busy, tenant-occupied core costs one scheduling quantum (~16 ms): a running tenant is never preempted mid-slice. Decode throughput saturates at gang width 8, so ceding cores past the knee is nearly free: elasticity auto-sizes the gang online.

大模型推理动态调度内核优化弹性资源

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。