Quick answer
Load shedding deliberately refuses work that a service cannot finish within its objectives. It protects useful throughput and latency for admitted work by preventing excess demand from becoming uncontrolled concurrency, queueing, pool contention, and retries. The design needs an explicit admission point, bounded waiting, priority rules, honest rejection responses, and evidence showing that the protected service remains inside its operating envelope.
Four controls are related but different. A rate limit bounds attempts over time, commonly by caller or quota. Admission control decides whether a specific unit of work may consume current capacity. Load shedding is the overload action that declines eligible work according to that policy. A concurrency limit bounds active work; an adaptive concurrency limit changes that bound from measured feedback, such as latency and completions, within reviewed minimum and maximum values. Adaptation does not remove the need for a hard safety ceiling.
Reject before allocating expensive resources. Preserve a protected priority class only when classification is server-controlled, and avoid letting background demand starve indefinitely. Use 429 Too Many Requests for a caller-specific rate or quota decision and 503 Service Unavailable for temporary service overload or unavailable capacity. Clients still need bounded, jittered retry policy and an end-to-end deadline. No status or response header proves that capacity has recovered.
Shared flash-sale incident stage
A promotion produces checkout overload. The edge accepts the burst, and order API saturation appears as rising in-flight requests and latency. Inventory calls wait for database connections, so inventory database pool wait rises before useful query execution. The system starts more requests than it can complete, while oversized queues make the initial dashboard look less alarming than the user experience.
Callers time out against a missing end-to-end deadline and retry at the edge, client library, and order service. This retry amplification raises offered load even after new shoppers slow down. Interactive checkout shares workers and inventory connections with reconciliation. Probe requests contend with business requests; failed readiness removes endpoints, and liveness restarts reduce warm capacity. Traffic concentrates on fewer replicas and makes the admission gap worse.
The immediate response is layered: pause the risky rollout and nonessential producers, disable redundant retry layers, reserve interactive capacity, and shed new work that cannot meet its budget. Rejecting early is safer than letting every request occupy a worker and then fail after payment or inventory work begins. Do not blindly enlarge the connection pool, because that can transfer collapse to the inventory database.
Recovery requires a controlled ramp. Drain only useful, unexpired backlog; add traffic or consumers in measured steps; and watch checkout success, tail latency, in-flight work, admission and rejection, oldest queue age, pool wait, retry volume, and probe transitions. A falling rejection rate is not sufficient if latency, old work, or database pressure remains high.
Core mechanism and evidence boundary
A static concurrency gate admits work while active permits are below a tested ceiling. An adaptive concurrency controller observes completed work and a pressure signal, then cautiously changes the limit within fixed bounds. It must distinguish a slow dependency from healthy spare capacity: increasing concurrency while service time rises can lower throughput. Use stable observation windows, limit adjustment size, preserve a hard maximum, and fall back to a conservative known value when telemetry is stale.
Rate limiting constrains a caller’s or tenant’s frequency or quota. Admission control makes the current capacity decision. Load shedding applies intentional rejection under pressure. Adaptive concurrency tunes active work rather than request frequency. These controls can coexist: a request can pass its quota yet be shed because the service is full.
For HTTP, 429 represents a client or caller rate limit or quota decision. 503 represents temporary overload, unavailable service, or insufficient current capacity. Retry-After is appropriate only when the server has a known, accurate, meaningful delay it can honor. Otherwise omit it and let clients use bounded backoff with jitter under the original deadline. Recovery remains an observed system state, not a promise embedded in metadata.
Record an admission rejection separately; it is not an application failure or business failure. Whether it counts against the user SLI depends on the service contract. Compare offered, admitted, rejected, and completed rates, plus in-flight work and latency. Sampled traces explain selected executions but cannot supply the concurrency denominator or the population rejection rate; use complete counters and gauges for those quantities.
The shared evidence model is deliberately bounded. route is a low-cardinality dimension containing a bounded route template such as /checkout, never a raw path. operation is a low-cardinality dimension for stable named work such as order.create. priority is a low-cardinality dimension for a bounded class or tier such as interactive. admission is a low-cardinality dimension for the accepted or denied capacity decision. rejection is a low-cardinality dimension for the denied-work decision, with reviewed values such as concurrency_limit.
deadline.remaining_ms is a teaching field measured in milliseconds. It is not a stable OpenTelemetry semantic convention; map it to the versioned telemetry contract actually deployed. Keep trace_id, request_id, and message_id as correlation fields in protected logs and traces. Never use metric labels or metric dimensions for trace_id, request_id, message_id, order_id, user_id, a raw URL, exception text, a credential, a token, or PII. Those values create unsafe cardinality, disclosure, or both.
Minimal reproducible implementation
The gate below uses a fixed safety ceiling and a separately evaluated adaptive target. Values are configuration inputs derived from load tests, not generic defaults.
handleCheckout(request, context):
remaining = clamp(context.timeRemaining(), 0ms, configuredMaximumBudget)
if remaining < minimumUsefulWork + cleanupReserve:
count(admission="denied", rejection="deadline_expired")
return deadlineResponse()
class = classifyOnServer(request.authenticatedPrincipal, request.route)
effectiveTarget = clamp(
adaptiveTargetFor(class),
configuredMinimumFor(class),
testedHardMaximumFor(class)
)
admissionWait = min(
configuredMaximumAdmissionWait,
remaining - minimumUsefulWork - cleanupReserve
)
permit = limiter[class].tryAcquire(limit=effectiveTarget, wait=admissionWait)
if permit is absent:
count(admission="denied", rejection="concurrency_limit")
return overloadResponse(status=503)
remainingAfterAdmission = clamp(
context.timeRemaining(),
0ms,
configuredMaximumBudget
)
if remainingAfterAdmission < minimumUsefulWork + cleanupReserve:
permit.release()
count(admission="denied", rejection="deadline_expired")
return deadlineResponse()
count(admission="accepted", rejection="none")
gaugeInFlight(class).increment()
started = monotonicNow()
try:
return createOrder(deadline=context.deadline, cancellation=context.cancellation)
finally:
gaugeInFlight(class).decrement()
permit.release()
controller.observe(
class=class,
duration=monotonicNow() - started,
completed=true,
telemetryFresh=metricsAreFresh()
)The admission boundary clamps the controller output independently, so a stale, corrupt, or racing target cannot exceed the tested hard maximum or fall below the reviewed minimum. It recomputes the trusted remaining deadline after any permit wait and releases the permit before rejecting work that no longer has minimum useful time plus cleanup reserve. The controller changes targets outside the request path. It considers latency relative to a tested baseline, completed throughput, dependency pool wait, and rejection. It ignores a window with insufficient completions or stale telemetry. A multiplicative decrease can react to sharp pressure; cautious additive increases reduce oscillation. The exact algorithm and window require workload-specific validation.
Return a quota decision differently from service saturation:
if callerQuota.exhausted():
return response(status=429, body={code: "CALLER_RATE_LIMITED"})
if serviceCapacity.unavailable():
return response(status=503, body={code: "TEMPORARILY_OVERLOADED"})Only include a delay header when a controller or scheduled maintenance window supplies a credible value. Do not derive it from one latency sample. Reject before database checkout, body expansion, or expensive authorization whenever security permits. If authentication is necessary to choose a tenant quota, keep authentication bounded and separately protected.
Failure modes and dangerous misconceptions
“Rate limiting solves overload.” A tenant can remain under quota while many tenants collectively exceed capacity, and one expensive request can consume more resources than many cheap requests. Pair quotas with current admission and resource controls.
“The highest throughput limit is best.” Near saturation, a small concurrency increase can sharply raise waiting without improving completions. Choose the operating point from useful throughput, tail latency, errors, and downstream pressure, leaving recovery headroom.
“Adaptive means autonomous and safe.” A noisy or delayed signal can cause oscillation. Bound the controller, version its configuration, require telemetry freshness, expose its current target, and retain a tested fixed fallback.
“Every overload response should invite immediate retry.” Synchronized retry turns rejection into renewed demand. Clients need one retry budget across layers, exponential backoff with jitter, idempotency, and sufficient remaining deadline.
“Priority lets checkout consume everything.” Absolute priority can starve maintenance, reconciliation, or health paths until they become incidents. Reserve explicit capacity and define fairness or minimum service for necessary classes.
“Autoscaling replaces shedding.” Scaling has observation and startup delay, may hit quotas, and can multiply load on a shared database. Immediate admission protects the interval before new safe capacity exists.
“A successful sampled trace proves the gate works.” Trace sampling can miss rejected or slow populations. Counters, gauges, and histograms establish rates and distributions; traces help explain representative cases.
Security/privacy/capacity/cost implications
Admission is part of the abuse boundary. Authenticate and validate cheaply, bound bodies and decompression, apply server-owned tenant and priority classification, and prevent forged headers from entering a protected lane. Separate public traffic from operational access, but audit any bypass and give it a narrow capacity reservation.
Rejection responses should reveal enough for a client to act without disclosing global capacity, other tenants, topology, or internal thresholds. Correlation identifiers belong in access-controlled telemetry. Avoid reflecting raw exception details or quota policy internals.
Reserved headroom and isolated pools cost money, while operating at the knee spends latency and error budget. Capacity planning must include the dependency bottleneck, failure of a replica or zone, retry load, controller reaction time, and recovery ramp. Adaptive limits are not a substitute for purchased capacity or query optimization. Telemetry cardinality also consumes memory and storage, so bounded dimensions are an operational constraint.
Testing and production validation
Find the safe envelope with representative payloads and service-time distributions. Increase offered load in steps while plotting completed throughput, p95 and p99 latency, in-flight work, queue age, inventory pool wait, CPU, memory, and rejection. The selected ceiling should remain safe after one replica disappears and should leave room for probes, cleanup, and reconciliation.
Test the controller against changes in work mix, not only changes in total arrival rate. A limit learned from cheap catalog reads can look healthy while expensive checkout calls starve. Build a matrix of operation, priority, payload size, cache state, and dependency latency; for each cell record offered and completed work, the active limit, permit wait, rejection reason, and downstream saturation. Then shift the mix during a run and verify the controller decreases safely without letting a fast class hide a slow one. Repeat with missing samples and delayed export. The conservative fallback should activate from telemetry freshness, not from a favorable stale value. Finally, replay the same demand with the controller fixed at its reviewed fallback so responders can distinguish adaptation behavior from the underlying admission gate.
Test bursts, sustained overload, a slow database, a partial network fault, cold starts, stale telemetry, and a controller restart. Verify that quotas return the intended outcome, capacity rejection happens before expensive work, priority reservations hold, and limit changes are bounded. Exercise clients that retry badly so the server remains protected. Confirm cancellation and deadlines do not imply rollback of a completed side effect.
Canary controller changes against a fixed cohort. Record configuration and rollout events, compare user SLI and resource evidence, and define automatic rollback for oscillation or pressure. During recovery, restore producers and limits gradually; confirm completed throughput improves without renewed pool wait, queue age, or probe churn.
Validate HTTP behavior with representative intermediaries and client libraries. Confirm that capacity and quota responses are not cached as successes, response bodies use stable machine-readable codes, and clients do not multiply attempts across SDK, proxy, and application layers. Simulate a caller that ignores backoff and prove the admission gate still bounds active work. Also test fairness with one heavy tenant, many small tenants, mixed request costs, and protected operational traffic. A controller that performs well under homogeneous requests may oscillate when service time changes by operation. Record the actual limit, completed throughput, and pressure at each load step so the selected envelope is reproducible rather than an unexplained production constant. Rehearse controller rollback while demand is high and verify the fixed fallback limit is immediately available.
Operations checklist
- Establish the checkout SLI, invariants, and tested sustainable capacity.
- Measure offered, admitted, rejected, completed, expired, and retried work.
- Keep an explicit hard concurrency ceiling around every adaptive target.
- Separate caller quota decisions from temporary service-capacity rejection.
- Use delay metadata only when the value is meaningful and operationally supportable.
- Preserve one deadline and retry budget across clients and services.
- Reserve server-controlled priority capacity without starving necessary background work.
- Alert on user impact; diagnose with in-flight, latency, queue age, pool wait, and probe transitions.
- Freeze adaptation when evidence is stale and use a conservative tested limit.
- Ramp recovery, validate the SLI, and reverse the step if pressure returns.
Official sources
- RFC 6585: 429 Too Many Requests
- RFC 9110: 503 Service Unavailable
- RFC 9110: Retry-After
- Google SRE: Handling Overload
- Prometheus instrumentation practices
Accessed 2026-08-02. HTTP semantics define protocol meaning; safe thresholds and adaptation behavior still require evidence from the deployed system.
Related reading
Place shedding inside Production Resilience for Backend Systems, preserve budgets with Timeouts, Deadlines, and Cancellation Propagation, and bound waiting with Backpressure, Bounded Queues, and Flow Control. The final Production Overload Troubleshooting runbook connects these controls to evidence. Continue through System Design or browse the Topics index.