"It works on my machine, but I have no idea why it's failing in production" is a symptom of an observability gap, not a debugging skill gap — you can't debug what you can't see, and most production incidents are resolved (or dragged out) based on how good your logging, metrics, and tracing actually are before the incident happens.
Observability is the ability to understand a system's internal state from its external outputs — logs, metrics, and traces are the three pillars. Logs record discrete events with context; metrics track numeric measurements over time (request rate, error rate, latency); traces follow a single request's path across multiple services, showing where time was actually spent.
Why Logging and Observability Matter (and When to Simplify)
Without structured, queryable observability data, debugging a production issue means guessing based on incomplete information — with good observability, you can ask specific questions ("which requests to this endpoint failed in the last hour, and what did they have in common?") and get answers, rather than reconstructing what happened from memory or sparse logs.
Simplify observability tooling for small, low-traffic applications where a full metrics/tracing stack is overkill — good structured logging alone often covers most debugging needs at small scale, and the operational overhead of running a full observability stack (Prometheus, Grafana, a tracing backend) isn't justified until the system's complexity actually requires it.
Getting Started with Logging and Observability
Structured logging — machine-parseable JSON rather than free-text strings:
logger.info("order_created", {
orderId: order.id,
userId: order.userId,
amount: order.total,
requestId: req.id,
});
A basic metric — counting and timing requests:
const timer = metrics.startTimer("http_request_duration_seconds");
await handleRequest(req, res);
timer({ route: req.route, status: res.statusCode });
Core Logging and Observability Concepts Every Developer Should Know
Structured logs (JSON, key-value) are queryable; free-text logs are not. A log line like "User 123 placed order 456" requires string parsing (fragile, error-prone) to extract data later; a structured log with { userId: 123, orderId: 456 } fields can be filtered, aggregated, and queried directly in your logging platform — this difference matters enormously at any real scale.
A correlation/request ID ties together everything that happened for a single request, across every log line, service, and trace span involved — without it, correlating "what happened for this specific failing request" across a distributed system is nearly impossible; with it, it's a straightforward filter.
Metrics answer "how much/how often" questions cheaply, at scale; logs answer "what exactly happened" questions with more detail but higher storage cost. Using metrics for aggregate trends (error rate over time) and logs for investigating specific incidents is the efficient division of labor — trying to answer every question from raw logs alone doesn't scale.
Distributed tracing shows where time is actually spent across service boundaries, which is often the fastest way to find the real bottleneck in a multi-service request — without tracing, "this request is slow" requires guessing which service is responsible; with tracing, the slow span is visible directly.
Common Logging and Observability Mistakes and How to Fix Them
Mistake 1: logging unstructured free-text strings, making logs hard to query and aggregate at scale. Fix: adopt structured (JSON) logging with consistent field names across the codebase.
Mistake 2: not propagating a request/correlation ID across service boundaries, losing the ability to trace a single request's full path through a distributed system. Fix: generate a request ID at the edge and propagate it through every downstream call and log line.
Mistake 3: logging too much or too little, either drowning signal in noise (logging every trivial event at info level) or missing critical context when something actually breaks. Fix: use log levels deliberately, and log enough context (relevant IDs, key state) at points that matter for debugging, without logging everything indiscriminately.
When Should You Add Distributed Tracing Instead of Relying on Logs Alone?
Add distributed tracing once your system involves multiple services handling a single request, and "which service is slow/failing" has become a real, recurring debugging question that logs alone answer poorly. Rely on structured logs alone for simpler systems (a monolith, or a small number of services) where the request path is short enough that correlation IDs and logs already give sufficient visibility without tracing's added infrastructure.
Logging and Observability in Production
Standardize structured logging with consistent field names and correlation IDs from the start of a project, since retrofitting this across an existing codebase with inconsistent logging conventions is a much larger effort than establishing it early. Also set up alerting on key metrics (error rate, latency percentiles) rather than relying on someone noticing a problem manually — observability data that nobody's watching doesn't actually help during an incident.
If your team currently debugs production issues primarily by grepping through unstructured log files, investing in structured logging with correlation IDs is a concrete, high-leverage improvement to make before the next incident.