Observability stack
Flask, PostgreSQL, OpenTelemetry Collector, Prometheus, Grafana, Loki, Tempo, Node Exporter and Alertmanager in a repeatable Docker lab.
← Back to hands-on labsSRE · OBSERVABILITY · RELIABILITY
A curated view of practical SRE work covering observability, database incidents, Kubernetes/EKS, Terraform reliability, recovery planning, and secure operations.
Application / PostgreSQL
↓
OpenTelemetry SDK
↓
OpenTelemetry Collector
↙ ↓ ↘
Prometheus Loki Tempo
\ | /
Grafana
↓
Alertmanager
↓
Incident channelFlask, PostgreSQL, OpenTelemetry Collector, Prometheus, Grafana, Loki, Tempo, Node Exporter and Alertmanager in a repeatable Docker lab.
Injected a controlled slow database query and correlated application latency, traces and PostgreSQL wait-state evidence.
Documented incident response, RTO/RPO, backup/restore, Multi-AZ, read replicas and recovery validation.
Covered Kubernetes workload failures, service discovery, Pod-to-RDS connectivity, IAM/IRSA and Terraform drift/state recovery.
Symptom: /db latency ≈ 3 seconds
↓
Prometheus detects elevated latency
↓
Tempo trace shows slow SELECT span
↓
pg_stat_activity shows PgSleep wait
↓
Query restored
↓
Latency and alert return to normalDetect → Alert → Acknowledge → Investigate → Mitigate → Verify → RCA
The troubleshooting path is followed end to end rather than stopping at a single component:
DNS → CDN/WAF → ALB → target group → Service → EndpointSlice → Pod → App → RDS