A benchmark identifies a strong configuration for a particular experiment. Deploying that configuration means bringing its assumptions along: how requests arrive, how long they live, where their cached state resides, and how quickly more capacity becomes usable.
Those assumptions can change while the hardware and server flags stay the same. A long document interrupts active streams. A follow-up lands on a cold worker. A burst ends before the new replica starts serving.
A serving configuration allocates limited resources under assumptions about the workload and system state. When those assumptions change, the bottleneck can move.
1. Read the benchmark before the number#
AgentX, MLPerf, and Artificial Analysis use different workloads to test inference performance. The useful question is whether those workloads resemble the service you want to run.
Shared context, tool waits, and different request lengths
Later requests depend on earlier responses
Request latency and throughput during replay
A defined model, dataset, and quality target
Requests arrive at random times or all at once
Throughput, with latency limits in Server
Set prompt lengths and simultaneous-request counts
Requests sampled over time
Time to first token and output tokens per second
AgentX replays traffic from coding-agent sessions. It preserves the pattern of prompts, follow-up turns, pauses, and parallel requests to test how a server handles that traffic. Later turns can depend on earlier responses. AgentX methodology
MLPerf compares systems using a defined workload and common rules. Its Server scenario sends requests over time and measures throughput while enforcing latency limits. Offline gives the system a batch of work and measures how quickly it finishes. MLPerf rules
Artificial Analysis tests hosted APIs. Its standard performance test uses set prompt lengths and concurrency levels to measure time to first token and output speed. These results describe the provider's endpoint as a user experiences it. API methodology
That difference matters in production. A server handling a steady supply of independent requests can behave differently when requests arrive in bursts or wait on earlier turns. Read the workload and arrival pattern alongside the headline result.
2. Meet the workload#
Imagine a shared online assistant. Some users are reading streamed answers when another submits a long document. Several requests keep generating long responses. A returning user asks a follow-up, while a group of application requests arrives together after tool calls finish.
The service routes and admits this work while earlier requests are still running. New inputs need prefill; active streams need repeated decode steps. Both use compute, and their retained contexts occupy KV cache. Some deployments also move that state between workers.
Pick a request. Follow it from arrival to streamed output.
Ready to follow a request. Play the journey, or advance one stage at a time.
A short prompt needs prefill before its first token. Each subsequent token still takes a decode step and keeps a sequence active.
This is an illustrative scene, not a claim that every product has the same traffic. Its useful property is that several demands overlap. Long contexts can correlate with long outputs. Tool completion can synchronize arrivals. Cache location can determine how much input work a request creates.
Matching average requests per second and average prompt length does not establish that this joint workload matches a benchmark. Nor does a median latency or a benchmark-specific gate establish the fraction of your requests meeting first-token, streaming, and completion targets together.
Quick glossary
Prefill processes an incoming prompt; decode generates subsequent tokens. KV cache stores attention keys and values associated with retained context. TTFT is time to the first received token; a reasoning model's first answer token may arrive later. Inter-token latency describes generation timing, but an average can hide one long pause. An SLO is a service-level objective, such as a first-token or streaming-latency target.
Artificial Analysis defines TTFT from request send to first received token, including a reasoning token when exposed. AgentX's reported ITL is a per-request average: (request latency − TTFT) / (output tokens − 1), for at least two output tokens. For a streaming product, record actual delivered-chunk gaps and first-answer time as well.
A configuration can remain appropriate when its operating conditions hold. DistServe first optimizes phase allocation and parallelism, then uses replication to meet traffic. It also monitors workload shifts and replans when the original allocation becomes suboptimal. The question is which conditions your deployment preserves.
3. Eight ways the bottleneck moves#
3.1 The batch disappears#
A configured maximum batch size is an upper bound, not a supply of requests. Suppose a tuning run keeps 128 sequences active on a replica, but only 16 are ready during a quiet production period. Keeping the maximum at 128 does not recreate the achieved batch.
The opposite appears during bursts. Evenly spaced requests and two clusters of requests can have the same window-average rate while demanding different amounts of immediate headroom. Try both patterns with the same ready capacity.
Eight requests in eight seconds. Watch two service slots handle them.
8 requests. 0 s maximum wait. Each request finds an available slot. This schedule creates no queue.
Continuous batching admits and retires sequences over time. Actual batch membership depends on arrivals, completion times, available memory, and admission decisions. Underfilled iterations can lose efficiencies from shared weight reads or larger matrix shapes. Holding requests to build a larger batch trades that opportunity against waiting time.
Scaling out under unchanged demand can reduce per-replica batching if the router spreads traffic evenly. Concentrating traffic may help occupancy while increasing queueing or changing cache placement. None of these outcomes follows from maximum batch size alone.
A test client can obscure the effect: a concurrency cap may delay sends when service slows. Preserve external arrival timing for independent demand, and retain dependencies for application turns. The vLLM benchmark CLI documents how concurrency limits affect actual request rate.
What to measure: inspect running sequences, scheduled tokens, queue time, and scheduled versus actual send times. Average GPU utilization cannot establish that the intended load reached the server.
3.2 A long prompt interrupts the stream#
Twenty-four conversations are decoding when a 32,000-token document arrives. In a shared scheduler that admits a large prefill before those decodes get another opportunity, the users see a gap in their streams. Aggregate token throughput can still look healthy.
Change the prefill chunking in the diagram and follow when decode gets another opportunity.
Watch a long prompt share a worker with an existing stream.
Play or step through the work. The stream advances at each D.
Decode waits through this large prefill step. Existing streams get their next opportunities after the prompt work finishes.
Sarathi-Serve divides prefill into chunks and admits decode work alongside them. The chunk budget limits how much prompt work can extend an iteration. Smaller chunks create more frequent scheduling opportunities, but can reduce prompt-processing efficiency or delay its completion. Repeated access to earlier KV state and kernel overhead also matter.
This is a scheduling mechanism, not a statement that every current engine handles an entire long prompt in one iteration. The figure is not a GPU execution trace, and chunking does not remove every source of interference. The appropriate budget depends on the active decodes, prompt distribution, and target hardware.
What to measure: align long-prompt arrivals with first-token delay and delivered-token gaps. Retune the prefill budget against both. Separate scheduling classes or prefill workers are alternatives when their isolation is worth the additional cost.
3.3 The same requests fill the KV cache#
Short fixed-length completions make memory demand predictable. Real requests can keep generating while others finish. Their retained contexts grow and remain live across many iterations, so a request-count limit that was safe for short completions can admit too much state.
Keep 48 requests active below and let some retain longer contexts. The count stays unchanged while the payload crosses the memory budget.
Same request count. A very different memory bill.
Turn short contexts into long ones. The active sequence count stays at 48.
Aggregate full-copy KV payload
32 GiB illustrative budget
Five long contexts fit at 31.5 GiB. The sixth takes the payload to 33 GiB.
What this model assumes
128 KiB of KV state per token, with no shared prefix, quantization, or eviction. Each long context replaces a short one, adding 1.5 GiB. The 32 GiB budget and payload describe one aggregate full copy, not memory per GPU. Model weights, activations, allocator overhead, and deployment sharding are excluded.
The illustrative model has 32 full-attention layers, eight KV heads, head dimension 128, and two-byte elements. Across one complete KV copy, each retained token requires 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB. Context includes the prompt and generated tokens retained so far. KV state and attention
Forty-eight contexts of 4,096 tokens consume 24 GiB. If sixteen grow to 16,384 tokens while the other thirty-two stay at 4,096, the payload becomes 48 GiB. A 32 GiB payload budget fits the first workload and not the second. This is aggregate complete-copy payload, not per-GPU memory under arbitrary sharding; it excludes sharing, compression, padding, and other allocations.
Under pressure, a scheduler may delay admission or preempt work. vLLM documents preemption and recomputation when KV space is insufficient. Paged allocation reduces waste; it cannot make unlimited live state fit.
What to measure: track retained-token distributions, KV occupancy, preemptions, and recomputation. Adjusting concurrency or separating unusually long requests can help; changing KV precision also requires a quality check.
3.4 The cache hit is on the wrong worker#
A conversation has a cached 20,000-token prefix and a 200-token follow-up. The worker holding that prefix can reuse it. A cold worker must process the prefix again unless a transfer or another cache tier supplies the state.
But the warm worker may have a queue. Change its wait and compare the two possible placements.
A cache hit is only useful if you can reach it in time.
Send a 20,000-token shared prefix plus 200 new tokens to one of two workers.
20,000 cached · 200 tokens to prefill
No cached prefix · 20,200 tokens to prefill
Queue + prefill only, at a hypothetical 10,000 tokens/s. Network, first-token decode, and other overhead are excluded.
A cached prefix saves 2 s of prefill here. A longer queue spends that saving.
What this model assumes
Modeled time = queue time + uncached prefill time, at a fixed hypothetical rate of 10,000 tokens/s. The cold worker has no queue. Cache lookup, network transit, first-token decode, transfer, and other overhead are excluded. Real prefill time is not linear across all workloads. These are model outputs, not benchmark results.
A cache hit saves computation without guaranteeing lower latency. Waiting can exceed the work saved. Dynamo's router combines prefix-overlap credit with worker load and can reduce that credit for busier workers. Neither strict affinity nor equal request counts fully describes this placement problem. The figure illustrates the tradeoff with assumed costs; it does not reproduce Dynamo's routing algorithm.
Reuse also requires the matching state to remain resident. A new replica can have free compute and none of the prefixes that made established workers efficient. Eviction or a change in routing can therefore increase uncached work without changing the visible request text.
Finally, prefix reuse avoids repeated prefill computation; full-attention decode still attends to the retained keys and values. A high hit rate can coexist with substantial decode work and KV occupancy.
What to measure: record uncached input work, queue time, and cache residency by worker. Revisit routing and cache capacity together when scale-out changes those distributions.
3.5 The phase split stops matching demand#
A service begins with long document summaries: fresh prompts and short answers. Later, users continue existing conversations and request longer explanations. Prefix reuse reduces new prefill work while longer outputs occupy decode workers for more steps.
The prefill pool can become underused as the decode queue grows. Explore how the phase demand and the allocation interact.
Change where eight worker units are assigned.
2 prefill : 6 decode, still 8 worker units. More units are assigned to token generation, with fewer left for prefill. The useful split depends on the workload.
Disaggregating prefill and decode removes some direct interference but creates separate queues. A ratio tuned for one workload does not rebalance itself. Doubling both pools may reduce queueing, but preserves the allocation ratio that no longer matches demand.
DistServe treats phase allocation and parallelism as optimization variables under latency requirements. Request counts alone do not determine each phase's GPU needs. A raw input/output token ratio is also insufficient: cached inputs alter prefill work, retained contexts alter attention work, and hardware processes the phases differently.
Reallocation has a transition cost. The Dynamo planner can change prefill and decode replica counts within a GPU budget, but new workers still need to become usable. Time spent in the wrong allocation matters alongside the eventual steady state.
What to measure: follow each pool's queue, service time, and utilization together with uncached prompt tokens and generation lifetimes. Test both the new allocation and the transition into it.
3.6 Communication becomes the limit#
An eight-GPU configuration is not fully described by “eight GPUs.” Tensor-parallel ranks exchange data during execution, making topology, collective latency, and available bandwidth part of the configuration. vLLM parallelism guidance
Prefill/decode separation adds another path: moving KV state between workers. Compute can remain available while that link is saturated.
Assume one decimal GB of state must move for every request.
80 Gb/s offered: 55 Gb/s above the link rate. This link cannot sustain that offered traffic, even before overhead.
At 1 decimal GB of KV transferred per request and ten requests per second, a shared path must carry 10 GB/s, or 80 Gbit/s, before overhead. A 25 Gbit/s path cannot sustain that demand. Actual transferred bytes depend on model architecture, input length, reuse, and implementation. DistServe makes placement depend on bandwidth.
Tensor parallelism has a related tradeoff. At small batches, local computation can shrink without eliminating synchronization overhead. If the model and working state fit, one eight-way replica and two four-way replicas are different uses of the same GPU budget. Two replicas duplicate weights and change batching and cache placement; one wider replica communicates across more ranks. Neither wins universally.
What to measure: separate collective time, KV-transfer bytes, transfer queues, and achieved bandwidth from kernel time. Transfers may overlap compute, so their full duration is not automatically added to TTFT. More workers behind the same saturated link can increase contention.
3.7 Speculation stops paying for itself#
Speculative decoding spends work proposing tokens so that one target-model verification can advance generation by several tokens. The return depends on how many tokens are committed and the cost of drafting and verification for the workload.
A setup tuned on repetitive replies may see lower draft–target agreement after the request mix changes. The domain label does not determine acceptance; the measured agreement does.
Extra work pays off only when enough tokens survive.
Four cheap draft tokens. One target verification. How much output gets committed?
Cost per committed token
normal decoding = 1×
Each round costs 1.4 units. It must commit more than 1.4 tokens to reduce cost per token.
What this model assumes
Four draft steps at 0.1 cost units each, plus one verification at 1 unit: (4 × 0.1 + 1) / g. Here g is average committed output per round, including a target or recovery token; it is not the draft acceptance rate. The range can reach 5 because four accepted draft tokens may be followed by a target token. Normal decoding costs 1 unit per committed token. Fixed illustrative costs exclude scheduling, memory, synchronization, and changing batch sizes; this is not an empirical speedup prediction.
In the illustrative model, four draft tokens each cost one tenth of a normal target step, and verification costs one target step. An attempt costs 1.4 target-step units. If it commits four tokens on average, the cost is 0.35 units per token. If it commits only 1.1, the cost rises to about 1.27, exceeding ordinary decoding.
“Committed” includes the correction or additional target token, not just accepted drafts. With four proposed tokens, an attempt in the original algorithm can produce one through five output tokens. The speculative-decoding paper separates draft cost and acceptance under explicit compute and verification-time assumptions.
The figure demonstrates a break-even threshold. Its assumed costs omit batching and scheduling effects and do not establish a hardware speedup. Verification need not keep the same cost as draft depth or serving load changes.
What to measure: inspect committed tokens per attempt and total time per committed token by request class. Tune draft depth, or disable speculation, when measured savings no longer cover its cost.
3.8 Capacity arrives after the burst#
A warm-worker benchmark excludes a transition that deployment must pay for. Conventional MLPerf measurement begins after system readiness. Production must load weights, initialize the worker, and finish required runtime warmup before using that capacity.
Move the readiness delay against the burst duration. A successful scaling event can still miss the requests that caused it.
A 20-second burst begins when a new replica is requested.
0 s of ready-capacity overlap with the burst. The burst has ended by the time the replica is ready. Existing queued work may remain.
If a burst lasts 20 seconds and requested capacity takes 60 seconds to become usable, reactive scale-out cannot supply that capacity during the burst. These are constructed timings, not typical startup-time claims. Useful cache state may develop later than service readiness.
ServerlessLLM treats checkpoint loading and startup-aware placement as scheduling concerns. Multiple workers can also contend while loading through shared storage or network paths.
What to measure: record detection-to-ready time and performance immediately after readiness. Warm reserve, predictive scaling, or admission control may be needed. Include reserved capacity in the cost comparison: output rate per busy GPU does not account for GPUs kept ready for a spike.
4. Test the operating point#
A benchmark winner is a candidate with useful evidence behind it. Turn that evidence into a deployment decision by checking the assumptions that your product is likely to change.
- Define success per request. Choose first-token or first-answer, streaming, and completion targets. Count requests that satisfy the applicable targets together; separate percentiles need not describe the same successful requests.
- Reproduce the candidate's conditions. Record model and quality settings, hardware, topology, engine version, workload, load generator, cache state, and readiness boundary. Confirm actual sends match the intended experiment.
- Perturb one mechanism at a time. Cluster arrivals; add long prompts to ongoing decodes; extend the output tail; send follow-ups to cold workers; shift the phase mix; exercise the real transfer and startup paths.
- Preserve the joint workload. After identifying individual effects, combine them in a representative trace. Keep correlations between context, output length, tool delays, and dependencies instead of matching averages alone.
- Account for the whole service. Include failures, rejections, unfinished requests, transitions, and ready spare capacity. Compare cost at matched quality and service requirements.
Changing one factor helps identify a cause. Combining realistic factors tests whether the resulting configuration still works when those causes interact. Keep both results: the controlled experiment explains the mechanism; the representative workload establishes its practical consequence.
The configuration may survive these tests. If it does, replication can be an effective next step. If it does not, the measurements identify which workload or deployment assumption requires a different allocation.
5. Sources and assumptions#
No new GPU benchmarks were run for this article. Worked serving examples, controls, and request-flow diagrams are illustrative. Benchmark presets are documented methodology details. Here, “fail” means missing the intended latency, capacity, or cost objective, even if responses remain correct.
The papers and official documentation support the mechanisms; they do not predict current-hardware performance. KV arithmetic assumes the stated full-attention cache and complete-copy payload. Routing, phase-allocation, queueing, and speculation examples use simplified stated costs or capacities. Neither their outputs nor historical paper results establish a ranking of current engines, GPUs, or providers.
Primary references:
- AgentX methodology and replay timing — trace structure, concurrency, and benchmark boundaries.
- MLPerf Inference rules — conventional Server/Offline scenarios and readiness.
- Artificial Analysis API methodology, AA-SLT, and AA-AgentPerf — distinct endpoint and system tests.
- Orca, OSDI 2022 — iteration-level scheduling and batching.
- Sarathi-Serve, OSDI 2024 — prefill/decode interference and chunking tradeoffs.
- Hugging Face cache explanation — KV state and retained context.
- vLLM optimization, prefix caching, parallelism, and benchmark CLI — memory pressure, reuse limits, topology, and load generation.
- Dynamo routing and planner — cache/load tradeoffs and phase allocation.
- DistServe, OSDI 2024 — phase placement, communication, replication, and replanning.
- Fast Inference from Transformers via Speculative Decoding, ICML 2023 — acceptance and draft-cost analysis.
- ServerlessLLM, OSDI 2024 — startup and checkpoint placement.