Load Balancers

Overview

A load balancer sits in front of a pool of servers and distributes incoming traffic across them, so no single server becomes a bottleneck or a single point of failure. It also continuously health-checks backends and routes around ones that fail, which is what turns a fleet of individually unreliable servers into a service with high overall availability. The two big design choices are which OSI layer to operate at (4 vs 7) and which algorithm decides where the next request goes.

Quick Reference

Syntax

# NGINX - Layer 7, least-connections, with health checks
upstream backend {
  least_conn;
  server 10.0.0.1:8080 max_fails=3 fail_timeout=30s;
  server 10.0.0.2:8080 max_fails=3 fail_timeout=30s;
  server 10.0.0.3:8080 weight=2; # gets roughly 2x the traffic
}

server {
  listen 80;
  location / {
    proxy_pass http://backend;
  }
}

Examples

# Path-based Layer 7 routing to different backend pools
location /api/ {
  proxy_pass http://api_backend;
}
location /static/ {
  proxy_pass http://static_backend;
}

# Kubernetes Service - a built-in L4 load balancer across matching pods
apiVersion: v1
kind: Service
metadata:
  name: my-app
spec:
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080
  type: LoadBalancer

Visual Diagram

Common Mistakes

• Running a single load balancer instance as a new single point of failure, production setups need the load balancer itself to be redundant (active-passive or active-active pairs, often via DNS or a floating IP).
• Health-checking a path that does not actually reflect service health (e.g. a static 200 OK with no dependency checks), so a server with a broken database connection still looks "healthy."
• Ignoring session affinity requirements, if application state lives in server memory rather than a shared store, routing a user's requests to different servers on each call breaks their session.
• Setting health-check thresholds too aggressively, so a single slow response flips a healthy server out of rotation and causes unnecessary churn.

Performance

• L4 load balancing has lower latency overhead than L7 since it does not parse request content, but it cannot make routing decisions based on that content.
• Connection draining (letting in-flight requests finish before removing a server from rotation during a deploy) prevents dropped requests during rolling updates.
• Sticky sessions (routing a client consistently to the same backend) trade even load distribution for state locality, prefer externalizing session state so any backend can serve any request, removing the need for stickiness entirely.

Best Practices

• Run load balancers in a redundant pair or managed, highly available service rather than a single instance.
• Use active health checks that exercise a real dependency path, not just a static liveness response.
• Prefer stateless backends with externalized session storage over sticky sessions, so load balancing stays simple and even.
• Match the algorithm to the workload: round robin for uniform cheap requests, least-connections for variable-duration requests, weighted routing for heterogeneous server capacity.