Average latency is a lie with good PR. Inference teams that ship good products optimize P99 — and, increasingly, the tail of the tail.
P99 math that matters
At 100 requests per second, P99 means one request in the worst second is slower than the target. If your SLA is "under 2 seconds," your P99 budget decides how much headroom you need at peak. Design for the percentile, then give it 20% slack for the day the load balancer hiccups.
Batching curves are your friend
Continuous batching turns idle GPU cycles into throughput, but the curve is not linear. There's a knee where latency grows faster than throughput — find that knee on your hardware, with your model, and pin the max batch size there. A good serving stack exposes these curves per endpoint because guessing is how P99 breaks.
The pre-production checklist
Before a model earns an endpoint: run a soak at peak load, verify autoscaling triggers before the queue grows, confirm cold-start time is inside your budget, and check that a single node failure doesn't spike the tail. If any of those fail, the fix is cheaper now than in an incident.
Autoscale to zero, then back
Idle replicas cost money; cold starts cost latency. Warm pools, pre-warmed engines, and scale-down grace periods make "zero" practical. The math is simple once you know your cold-start time — which is why it should be measured for every model before it earns an endpoint.