A conventional limiter charges one unit per request and decides once, on arrival. For LLM inference neither half holds: cost is tokens, varying by three orders of magnitude between a one-line completion and a 200k-token context — and the larger half of that cost, the output, is not known until generation stops.
Refill, compare, deduct and lease happen inside a single Lua script, so N gateway replicas cannot over-admit by a factor of N. Redis runs a script to completion, which serialises the whole decision on the shard without any lock the gateway has to hold, renew, or lose.
Deduct an estimate, write a lease with a deadline, return a reservation id.
Under worst_case the estimate is the declared ceiling; under
adaptive it is an EWMA of what this model has actually been producing.
The service has a third mode, quantile, which holds the q-th quantile
of a rolling histogram instead — so the knob states the intended overrun rate
rather than leaving it to a safety factor.
Generation finished, so the real cost is known. Refund the difference, or charge the overrun. Idempotent, because this call happens right after a slow generation — exactly when clients time out and retry.
A client that dies between the two calls would otherwise burn its tenant’s budget forever, with every individual decision still correct and nothing to alert on. Expired leases are swept in bounded batches, on the admission path.
Send requests and watch the budget split into free, held and spent. Complete one to see the hold reconciled. Abandon one to simulate a client crash — the budget stays held until the lease expires, and the next admission reclaims it.
src/llm/admission-scripts.ts — the same constraint ordering,
fractional token-bucket refill, bounded inline lease sweep, idempotent settlement and
EWMA estimator. It is not the service: there is no Redis, no PostgreSQL and no
network here, so nothing on this page is a latency measurement. Real numbers are
below, and they come from committed artifacts.
Two deliberate differences. The lease default is 8 s rather than 120 s and the
budget is scaled down, both so that expiry and refill are observable at human speed.
And every request here is interactive at the default
batchFloor = 0, where the three priority-floor conditions in the
real script reduce exactly to the hard gates above them and are therefore unreachable
— so the ladder below is the production ladder for this configuration, not a
simplification of it. The service has a third reservation mode,
quantile, which this page does not offer: it needs a histogram with
enough settled completions behind it to mean anything, which is more traffic than a
page you click through produces.
An earlier version of this page also noted that the simulator applied refill on commit
while the production Lua did not — that defect has since been
fixed,
with a regression test that fails against the old script, so the two now agree.
Open-loop load against absolute arrival deadlines, full reserve + commit cycle per
request, budgets set far above the offered rate so this measures the cost of
deciding rather than the cost of refusing. Raw output for every row is committed
under bench/results/.
| offered cycles/s | achieved | p50 ms | p95 ms | p99 ms | p99 range | spread | gateway CPU | Redis CPU |
|---|---|---|---|---|---|---|---|---|
| 250 | 250 | 0.41 | 0.51 | 0.73 | 0.60–0.85 | 1.4× | 15.0% | 5.0% |
| 500 | 500 | 0.38 | 0.50 | 0.61 | 0.57–0.98 | 1.7× | 21.9% | 8.7% |
| 1000 | 1000 | 0.41 | 0.57 | 0.73 | 0.68–0.74 | 1.1× | 33.7% | 15.3% |
| 1500 | 1500 | 0.45 | 0.68 | 1.22 | 0.87–1.53 | 1.8× | 45.4% | 21.4% |
| 2000 | 2000 | 0.59 | 0.85 | 1.47 | 1.20–2.08 | 1.7× | 55.8% | 27.5% |
| 2500 | 2499 | 0.66 | 1.08 | 2.41 | 1.64–13.36 | 8.1× | 66.7% | 33.1% |
| 3000 | 2999 | 0.70 | 1.83 | 6.93 | 2.45–12.42 | 5.1× | 75.7% | 37.0% |
Median of five sweeps, with the range and the max/min spread beside it. 32 cores,
Node 26.7.0, Redis 8.10.1 and PostgreSQL 18.6 over loopback; host load average 1.3–3.4,
the quietest of the three runs, recorded in bench/results/repeat/load.txt. Reproduce with
./bench/repeat.sh 5 20. The CPU columns are from a single sweep rather than the median of five, because CPU
repeats closely between runs where the tail does not, and the harness writes them
into each per-rate artifact, so they are sourced like every other number here.
Those two columns used to be wrong. The harness computed
ticks / CLK_TCK / wall * 100 in bc, which truncates after
every division — so at a 23-second run the answer was quantised to steps
of ten and each step rounded down. The published column read 10/20/30/40/50/60/70 at
seven consecutive rates, which looks like a textbook linear fit and was an artefact of
the arithmetic, and Redis read 0% while using 5% of a core. Multiplying before
dividing, once, gives the figures above. The conclusion is unchanged — linear in
offered rate, Redis well under half a core — and the numbers are five to ten
points higher.
| offered cycles/s | p50 single | p50 cluster | p95 single | p95 cluster | gw CPU single | gw cluster | redis single | redis cluster |
|---|---|---|---|---|---|---|---|---|
| 250 | 0.41 | 0.44 | 0.51 | 0.53 | 15.0% | 15.9% | 5.0% | 9.0% |
| 500 | 0.38 | 0.41 | 0.50 | 0.52 | 21.9% | 23.0% | 8.7% | 16.0% |
| 1000 | 0.41 | 0.43 | 0.57 | 0.61 | 33.7% | 36.7% | 15.3% | 28.4% |
| 1500 | 0.45 | 0.51 | 0.68 | 0.76 | 45.4% | 50.3% | 21.4% | 39.8% |
| 2000 | 0.59 | 0.60 | 0.85 | 0.92 | 55.8% | 62.1% | 27.5% | 50.8% |
| 2500 | 0.66 | 0.65 | 1.08 | 1.21 | 66.7% | 71.5% | 33.1% | 59.0% |
| 3000 | 0.70 | 0.58 | 1.83 | 1.58 | 75.7% | 84.6% | 37.0% | 70.3% |
Three masters, three replicas, 24 tenants — and the tenant count is the experiment. Every key of one pair shares a hash tag, so one tenant is one slot is one master; a cluster run driven by a single tenant measures one shard and reports a reassuring number for the wrong reason. The harness refuses to run if the tenants share a master. p50 moves by 0.02–0.06 ms, because ioredis caches the slot map and steady-state routing is a CRC16 and a table lookup rather than a round trip. p99 is deliberately absent: three runs have shown the tail to be a property of this host, so comparing one cluster sweep against a five-sweep median would be comparing noise and calling it a routing cost.
Redis's 1.9× is replication rather than sharding. Three masters with no replicas use 5.4% of a core at 250 cycles/s against single-node's 5.0% — sharding splits the work rather than adding to it — and adding three replicas takes it to 9.0%, about what applying the same write stream twice costs. Gossip is not it: six idle nodes use 0.9% of one core between them. Not measured: failover. Everything here ran against a healthy cluster.
| mode | admitted | mean held | unused | overrun | target |
|---|---|---|---|---|---|
| worst_case | 75 | 5341 | 72.5% | 0% | 0% |
| adaptive | 231 | 1734 | 12.9% | 12.6% | — |
| quantile | 212 | 1889 | 19.8% | 6.1% | 10% |
400 requests arriving together against 400,000 tokens/minute. Both estimates admit
about 3× as much of the burst as reserving the ceiling.
quantile holds 9% more per request than adaptive, admits
8% fewer, and halves the overrun rate. Overruns are charged at commit, so they are
paid for rather than escaping the budget.
The target column is the point. A safety factor of 1.5 cannot be wrong,
because it never claimed anything; q = 0.90 asks for a 10%
overrun rate and can be held to it. It is not: the run delivered 6.1%, undershooting
for two reasons in the design rather than the data — the histogram holds the
upper edge of its bucket, up to 19% above the quantile, and the estimate comes
from a finite recent window. Interpretable and directionally right; not calibrated.
| assumed output median | unused | worst | adaptive | quantile | adapt ovr | quant ovr |
|---|---|---|---|---|---|---|
| 90 | 74.3% | 75 | 266 | 255 | 12.8% | 6.7% |
| 180 | 72.0% | 75 | 230 | 212 | 12.6% | 6.1% |
| 400 | 66.2% | 75 | 174 | 146 | 12.1% | 2.7% |
| 900 | 54.0% | 75 | 112 | 85 | 9.8% | 3.5% |
| 2000 | 35.0% | 75 | 74 | 74 | 0% | 0% |
The 72% is an input, not a finding — arithmetic on a log-normal chosen
to resemble chat traffic, against a fixed 4096 ceiling. No production trace was
available, so the result is reported as a sweep over the assumption it rests on.
The last row is the honest reading: once callers declare close to what they use,
both estimates collapse onto worst_case — the hold clamps to the
declared ceiling — and the overrun risk is not worth carrying. The last column
is the second one: quantile sits below its 10% target at every point,
by a margin that widens as the distribution tightens.
worst_case admitting exactly 75 at every median is the self-consistency
check: that mode reserves the declared ceiling and ignores what is generated, so its
count cannot depend on the output distribution. An earlier version of this benchmark
showed it varying from 42 down to 0 — which is how the calibration phase was
found to be draining the budget before the measured burst began.
| floor | class | offered | admitted | admitted % |
|---|---|---|---|---|
| 0 | interactive | 221 | 94 | 42.5% |
| 0 | batch | 6362 | 3085 | 48.5% |
| 0.25 | interactive | 221 | 221 | 100.0% |
| 0.25 | batch | 6361 | 2834 | 44.6% |
One tenant, one budget. Batch offers 3× what the budget sustains, interactive a tenth of it. Both phases replay the same drawn traffic. Interactive goes from 42.5% to 100%; batch pays 3.9 points for it, because the floor is a level rather than a rate — everything refilling above it is still batch's to take. Total work moves from 158.9 to 152.7 admissions/s.
| floor | admitted/s | tokens charged/s | % of refill |
|---|---|---|---|
| 0 | 158.9 | 149,717 | 99.8% |
| 0.25 | 152.7 | 149,738 | 99.8% |
The last column is the check that anything was contended at all, and it is not decoration. The first version of this benchmark reported 100% for both classes under both floors — true, and meaningless. A per-minute bucket starts full, and at overload factor X it takes 60/(X−1) seconds to drain, whatever the size of the budget: 29 seconds here, against a 20-second phase. Each phase now drains to its own steady state — which for a floored class is the floor itself — before anything is recorded.
The run admits ~159 req/s against a size-blind estimate of 106 (budget ÷ mean request). Both are right: admission from a contended bucket is not size-blind, because a large request needs more budget present at the instant it arrives, so the admitted set skews small (mean 989 tokens charged against a 1,445-token offered mean). That is a property of a token bucket, not of the floor. Note that the floor removes it for interactive traffic: at 100% admission its mean charged is 1,446, its own offered mean. Large interactive requests stop being the ones that lose.
A sweeper is the obvious design and the wrong one: another process to deploy, monitor and elect a leader for, returning budget on its own schedule instead of when someone needs it — and it introduces the one race the protocol most needs to avoid, refunding a hold at the moment its owner commits it. A bounded sweep at the top of the admit script has none of those properties and needs no coordination.
An overrun is charged as a negative refund and the tenant refills out of the hole. Clamping at zero would make tokens beyond the reservation free, turning underestimation into a strategy and making adaptive mode exploitable by design rather than merely imprecise.
Commit is called by a client that just waited out a slow generation, which is exactly when its request times out and gets retried. A retried reserve, by contrast, is a genuinely new request for capacity — the first is released by its lease, and deduplicating it would need an idempotency key the caller has no reason to have.
The generic limiter fails open; LLM admission fails closed. A limiter that returns 500 on a Redis blip has made its own availability the ceiling on everything it fronts. But unmetered traffic into a fixed accelerator pool does not degrade gracefully — it queues until the backend collapses and every in-flight request is lost, not just the excess.