DUPATANE← Back to hands-on labs

SRE · OBSERVABILITY · RELIABILITY

Investigating reliability across the stack

A curated view of practical SRE work covering observability, database incidents, Kubernetes/EKS, Terraform reliability, recovery planning, and secure operations.

OpenTelemetryPrometheusGrafanaPostgreSQLKubernetes / EKSTerraform

Architecture

Application / PostgreSQL
        ↓
OpenTelemetry SDK
        ↓
OpenTelemetry Collector
   ↙        ↓        ↘
Prometheus  Loki    Tempo
      \      |      /
          Grafana
             ↓
      Alertmanager
             ↓
     Incident channel

What was built and tested

01

Observability stack

Flask, PostgreSQL, OpenTelemetry Collector, Prometheus, Grafana, Loki, Tempo, Node Exporter and Alertmanager in a repeatable Docker lab.

02

Incident simulation

Injected a controlled slow database query and correlated application latency, traces and PostgreSQL wait-state evidence.

03

Reliability operations

Documented incident response, RTO/RPO, backup/restore, Multi-AZ, read replicas and recovery validation.

04

Platform troubleshooting

Covered Kubernetes workload failures, service discovery, Pod-to-RDS connectivity, IAM/IRSA and Terraform drift/state recovery.

Incident case study

Symptom: /db latency ≈ 3 seconds
        ↓
Prometheus detects elevated latency
        ↓
Tempo trace shows slow SELECT span
        ↓
pg_stat_activity shows PgSleep wait
        ↓
Query restored
        ↓
Latency and alert return to normal
Recovery was validated using the same signals that detected the incident: application latency, metrics, traces and database activity.

Operational approach

Detect → Alert → Acknowledge → Investigate → Mitigate → Verify → RCA

The troubleshooting path is followed end to end rather than stopping at a single component:

DNS → CDN/WAF → ALB → target group → Service → EndpointSlice → Pod → App → RDS

Engineering outcomes