Service Mesh Basics

Overview

A service mesh adds a uniform layer of traffic control, security, and observability across every service-to-service call in a cluster, without touching application code. It works by injecting a sidecar proxy into every Pod that transparently intercepts all network traffic in and out of the application container, so capabilities like mutual TLS, retries, and per-request metrics apply identically everywhere, instead of being reimplemented (and inevitably drifting) inside every individual service. The trade-off is real: another control plane to operate, and a proxy hop added to every request. Meshes tend to make sense once the number of services and the need for consistent cross-cutting behavior across them outgrows what's manageable to implement per-service.

Quick Reference

Syntax

# Istio VirtualService - shift 10% of traffic to a canary version
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: api
spec:
  hosts: [api]
  http:
    - route:
        - destination: { host: api, subset: stable }
          weight: 90
        - destination: { host: api, subset: canary }
          weight: 10

Examples

# The application container is unaware of the mesh entirely;
# it just makes a normal call to another service's DNS name.
# The sidecar proxy transparently intercepts it, applies mTLS,
# retries, and timeout policy, and records latency/error metrics.
curl http://inventory-service.default.svc.cluster.local/check

Visual Diagram

Common Mistakes

• Adopting a service mesh for a small number of services "because it's best practice", the operational overhead (another control plane, proxy upgrades, debugging an extra network hop) often isn't justified until the service count and cross-cutting-concern pain both grow.
• Forgetting that sidecar proxies consume real CPU/memory per Pod, at cluster scale, this is a meaningful, easy-to-underestimate capacity cost.
• Debugging a networking issue without realizing a sidecar proxy is in the path, mesh-injected retries and timeouts can mask or reshape failures in ways that look like application bugs.
• Enabling strict mTLS cluster-wide before migrating every service, breaking any service that isn't yet mesh-injected and expects plaintext traffic.

Performance

• Each sidecar hop adds latency, typically low single-digit milliseconds per hop with a well-tuned proxy, but it's a real, cumulative cost across a request that fans out to multiple services.
• Proxy resource consumption scales with Pod count, not request volume alone, every Pod gets its own sidecar, so cluster-wide CPU/memory overhead grows linearly with the number of Pods, not just traffic.
• Automatic retries configured at the mesh level can amplify load during an incident (a slow downstream service getting retried by every caller) if retry budgets aren't configured carefully.

Best Practices

• Adopt a mesh incrementally, namespace by namespace or service by service, rather than cluster-wide on day one.
• Set explicit retry budgets and circuit-breaking thresholds; the defaults are rarely right for every service's actual failure characteristics.
• Treat mesh-provided mTLS and observability as the reason to adopt one; if you only need basic load balancing, a plain Kubernetes Service is enough.
• Monitor the sidecar proxies themselves (their own resource usage and error rates), not just the applications behind them; the proxy is now part of the critical path.