Health Checks & Observability
Master the three pillars of observability (Metrics, Logs, Traces), Kubernetes liveness and readiness probes, distributed tracing with OpenTelemetry, alerting strategies, and how to build dashboards that catch issues before users do.
Master the three pillars of observability (Metrics, Logs, Traces), Kubernetes liveness and readiness probes, distributed tracing with OpenTelemetry, alerting strategies, and how to build dashboards that catch issues before users do. This hands-on tutorial focuses on practical implementation of health checks & observability concepts.
Health Checks & Observability
Observability is the ability to understand the internal state of a system from its external outputs. You can't fix what you can't see. In distributed systems with dozens of microservices, a request might touch 10 services before returning an error — without proper observability, you have no idea which service caused it.
[!IMPORTANT] Monitoring = knowing that something is broken. Observability = understanding why it's broken. You need both. Monitoring tells you "error rate is 5%." Observability helps you trace which service, which query, which user is causing it.
Health Check Endpoints
Every production service must expose health endpoints for load balancers and orchestrators (Kubernetes) to query:
The Three Pillars of Observability
The Four Golden Signals (Google SRE)
Google's SRE book defines four signals that are sufficient to monitor any service:
| Signal | Metric | Alert Threshold Example |
|---|---|---|
| Latency | p99 response time | Alert if p99 > 500ms for 5 min |
| Traffic | Requests per second | Alert if drops >50% (possible outage) |
| Errors | Error rate (4xx/5xx %) | Alert if >1% error rate |
| Saturation | CPU/Memory/DB connections | Alert if DB connections >80% of pool |
Quiz
Quiz
Question 1 of 2Your service's database goes down. Your liveness probe (/health/live) returns 503. Your readiness probe (/health/ready) also returns 503. What does Kubernetes do?
Key Takeaways
✅ Liveness probe failure = restart the container. Readiness probe failure = remove from load balancer only.
✅ The three pillars: Metrics (what's happening), Logs (what happened), Traces (why it happened).
✅ Structured JSON logs are machine-queryable — always include trace_id, user_id, duration_ms.
✅ Distributed tracing propagates a trace_id across services, enabling end-to-end request visibility.
✅ Monitor the four golden signals: Latency, Traffic, Errors, Saturation.
Official Resources
- 📖 OpenTelemetry Documentation — Standard for metrics, logs, traces
- 📖 Prometheus Documentation — Metrics collection & alerting
- 📖 Jaeger Tracing — Open-source distributed tracing
- 📖 Google SRE Book — Monitoring — Four golden signals
What's Next?
Your infrastructure is resilient and observable. Now let's tackle async communication between services.
Next: Message Queues — Kafka, RabbitMQ & SQS — How async messaging decouples services and enables event-driven architectures.