What you will take away
- Correlate logs, metrics and traces with one request id or none of it helps.
- Alert on user-visible symptoms; keep resource metrics for diagnosis.
- Tail-based sampling keeps the slow and failing requests you actually need.
The question observability has to answer
Not "is the server up" but "why was this request slow for this customer at this time". In a single application the stack trace answers it. Across a dozen services and a queue, no single log file contains the story, and the investigation becomes an exercise in correlating timestamps by hand.
The three signals each answer a different question: metrics say something is wrong and when it started, traces say where in the path, and logs say what exactly happened at that point.
Structured logs, and the one field that matters
Log as structured events with consistent field names - a trace id, a service name, an operation, a duration and an outcome - not as prose. Prose is readable by one engineer and queryable by none.
The critical field is the correlation id, propagated across every service and queue hop. Without it, you have three expensive tools that cannot be joined. With it, one identifier moves you from a metric spike to the exact log line.
Logs, metrics and traces that cannot be joined by a single request id are three bills, not one platform.
Metrics worth collecting
- For every service: request rate, error rate and duration distribution, per endpoint.
- For every resource: utilisation, saturation and errors - queue depth and connection pool usage included.
- A small number of business metrics, such as orders placed per minute, which detect failures no technical metric catches.
- Percentiles rather than averages. An average hides the tail, and the tail is what users complain about.
Tracing and sampling
A trace shows one request as a tree of spans across services, which turns "the checkout is slow" into "the tax service is called four times sequentially and each call takes 300ms". That is the difference between a hypothesis and a fix.
Full tracing at high volume is costly, so sample - but prefer tail-based sampling, which decides after the request completes and can therefore keep everything that was slow or failed. Head-based sampling discards the interesting requests at random.
Alert on symptoms, not on causes
Page a human for things users can feel: elevated error rate, latency past the objective, a queue growing without draining, a business metric dropping off its normal curve. High CPU is not an incident; it is a clue.
Every alert needs an owner and a runbook, and an alert that fires regularly without action must be fixed or deleted. Alert fatigue is the mechanism by which a well-instrumented system still fails unnoticed.
OrbixEdge Engineering
The engineers who build and run the systems we write about - platform, cloud and product teams shipping for clients in retail, health, logistics and fintech.
