Load Balancing Explained

Quick answer

Load balancing distributes incoming traffic across multiple backend instances. It helps improve availability, scale capacity, and avoid overloading a single server.

A load balancer can run at different layers, from DNS and TCP to HTTP-aware routing. The right design depends on traffic shape, failure behavior, session needs, and deployment model.

Why load balancing matters

One backend server is a single point of failure and a capacity limit. If it crashes, the service goes down. If traffic grows beyond its capacity, latency and errors rise.

With multiple instances, traffic can be spread out:

client -> load balancer -> app server A
                      -> app server B
                      -> app server C

The load balancer becomes the entry point that decides where each request should go.

Common algorithms

Algorithm	How it works	Good for
Round robin	Sends requests to servers in order.	Similar servers with similar request cost.
Least connections	Sends traffic to the server with fewer active connections.	Long-lived or uneven requests.
Weighted routing	Sends more traffic to stronger or preferred servers.	Mixed instance sizes or canary rollout.
Hash-based routing	Uses a key such as client IP or session ID.	Keeping related requests on the same backend.

Round robin is simple, but real traffic is rarely perfectly even. Some requests are heavier than others, and some servers may be slower.

Health checks

A load balancer should avoid sending traffic to unhealthy instances.

Health checks can be simple:

GET /health
200 OK

But a good health check should reflect whether the instance can actually serve traffic. If the app is running but cannot reach a required database, returning healthy may be misleading.

Avoid making health checks too expensive. A health endpoint that performs heavy database work every second can create load by itself.

Layer 4 vs Layer 7

Layer 4 load balancing works at the transport level, commonly TCP or UDP. It is fast and does not need to understand HTTP details.

Layer 7 load balancing understands application protocols such as HTTP. It can route based on host, path, headers, cookies, or request content.

Examples:

/api/orders -> order service
/api/users  -> user service

Layer 7 routing is powerful, but it adds more application-aware configuration.

Sticky sessions

Sticky sessions send repeated requests from the same client to the same backend instance.

This can be useful when session state is stored in memory on the app server. But in-memory session state makes scaling and failover harder. If that instance goes away, the session may be lost.

For many web APIs, it is better to store session state in a shared store or use stateless authentication where appropriate. Read JWT vs Session Authentication for the tradeoffs.

Load balancing and deployments

Load balancers are useful during deployments:

• Add new instances and wait for health checks.
• Gradually route traffic to a new version.
• Remove old instances from rotation.
• Drain connections before shutdown.
• Roll back by routing traffic away from bad instances.

Weighted routing can support canary deployments by sending a small percentage of traffic to a new version first.

Failure modes

Load balancing does not remove every risk.

Common failure modes:

• Health checks are too shallow and mark broken instances healthy.
• All instances depend on the same failing database.
• Sticky sessions keep too much traffic on one node.
• Retry storms multiply traffic through the load balancer.
• The load balancer itself is not highly available.

Pair load balancing with timeouts, circuit breakers, rate limiting, and observability.

Health check design

A health check should answer a specific question: can this instance safely receive traffic right now? That is different from “is the process running?” A process can be alive while its thread pool is exhausted, its database connections are broken, or its configuration is invalid.

Many systems use separate endpoints:

• Liveness: the process is running and should not be restarted.
• Readiness: the instance is ready to receive traffic.
• Dependency health: detailed checks for operators and dashboards.

Keep readiness checks fast and predictable. If every load balancer probe performs expensive database work, health checks can become their own source of load. A common compromise is to check critical local state frequently and deeper dependencies through background health signals.

Connection draining

Deployments and scale-down events need connection draining. When an instance is removed from rotation, the load balancer should stop sending new requests while allowing in-flight requests to finish for a short grace period.

Without draining, users may see errors during every deployment even if the new version is healthy. Draining matters for uploads, long API requests, streaming responses, and WebSocket-like connections. The application also needs graceful shutdown behavior: stop accepting new work, finish or cancel in-flight work safely, and release resources.

Load balancing is therefore both a traffic distribution feature and a deployment safety feature. Treat it as part of the release process, not only the networking layer.

Common mistakes

The first mistake is assuming load balancing fixes slow database queries. If all app servers wait on the same overloaded database, adding more app servers may increase pressure.

The second mistake is using sticky sessions as a default instead of designing session storage intentionally.

The third mistake is not draining instances before shutdown. Cutting traffic instantly can break in-flight requests.

The fourth mistake is ignoring request cost. One heavy export request may not equal one simple read request.

Related reading

Read Circuit Breaker Pattern Explained for dependency failure protection and Rate Limiting in APIs Explained for controlling inbound traffic. For stateful authentication tradeoffs behind sticky sessions, read JWT vs Session Authentication. You can also browse the System Design topic cluster.