Overview
Observability is what lets an engineer go from "something is wrong" to "here is exactly why" using only the signals a system already emits, without deploying new debugging code in the middle of an incident. It rests on three complementary signal types, logs, metrics, and traces, each answering a different question, and each with different cost and granularity trade-offs. Teams that only have dashboards ("monitoring") can detect that something broke; teams with real observability can also reconstruct why, for failure modes nobody explicitly built a dashboard for in advance.
Quick Reference
| Signal | Answers | Granularity | Storage cost |
|---|---|---|---|
| Metrics | What's happening in aggregate, over time | Low (pre-aggregated) | Low |
| Logs | What exactly happened in one event | High (full detail) | High |
| Traces | Where did time go across a distributed request | High (per-request, per-span) | Medium-high |
Syntax
// Metrics - cheap, aggregate, low-cardinality labels
httpRequestsTotal.inc({ route: '/api/orders', method: 'POST', status: 201 })
// Structured logs - high detail, correlated by a request id
logger.info('order created', { requestId, orderId, userId, amountCents })
// Traces - spans correlated across service boundaries
const span = tracer.startSpan('create-order')
span.setAttribute('order.id', orderId)
span.end()Examples
// Propagating a trace/request id across a service boundary is what
// makes correlation between logs, metrics, and traces possible at all.
async function handleRequest(req) {
const requestId = req.headers['x-request-id'] ?? crypto.randomUUID()
logger.info('request received', { requestId, path: req.path })
const res = await fetch('http://inventory-service/check', {
headers: { 'x-request-id': requestId },
})
return res
}Watch metric label cardinality
A label like user_id or request_id on a metric turns into millions of stored time series. Keep metric labels low-cardinality (route, method, status code) and put high-cardinality detail in logs or trace attributes instead.
Visual Diagram
Common Mistakes
- Adding a high-cardinality label (user ID, request ID, raw URL with query params) directly to a metric, silently degrading or crashing the metrics backend under real traffic.
- Logging without a shared correlation id, making it impossible to reconstruct one request's full path across multiple services during an incident.
- Building dashboards only for known failure modes and calling it "observability", that's monitoring; real observability is measured by how well unknown failure modes can still be debugged.
- Alerting on symptoms with no accompanying context (a bare "error rate high" page) instead of including enough signal (which route, which dependency) to start debugging immediately.
Performance
- Metrics are cheap specifically because they're pre-aggregated at collection time; the storage cost is proportional to cardinality, not request volume, which is why cardinality control matters so much.
- Full-detail logging and tracing on every request gets expensive at scale, sampling (e.g., trace 1% of requests, always trace slow/error requests) keeps cost bounded without losing the signal that matters most.
- Synchronous, blocking telemetry calls (logging or metric emission that waits on a network round-trip) add real latency to the request path, telemetry should be buffered/batched and shipped asynchronously.
Best Practices
- Propagate a single request/trace id through every service a request touches, and include it in every log line and every span.
- Keep metric labels low-cardinality; push anything with unbounded unique values into logs or trace attributes.
- Instrument for the questions you don't yet know you'll need to ask, not just the dashboards you already have, structured logs with rich, queryable fields age much better than plain text messages.
- Alert on symptoms that indicate user-facing impact (latency, error rate) rather than every internal metric crossing a threshold, to avoid alert fatigue.