Observability Stack Architecture Prometheus Grafana

The Problem This Blueprint Solves

Your team has CloudWatch dashboards for some services, Datadog for others, application logs go to CloudWatch Logs without structure, and distributed tracing does not exist. When an incident occurs, engineers spend 45 minutes correlating logs, metrics, and traces across 5 different tools before they can even identify which service is at fault. Your MTTR is 2.3 hours, and 60% of that is detection and diagnosis time — not remediation.

This blueprint is the observability platform I built for an e-commerce company running 67 microservices, reducing MTTR from 2.1 hours to 18 minutes by implementing unified telemetry collection, correlated dashboards, and automated anomaly detection that identifies the failing service before a human opens a laptop.

What You Get

• Architecture diagrams — Telemetry collection pipeline (metrics, logs, traces), data routing and storage topology, dashboard hierarchy, and alerting escalation flow (Draw.io)
• Terraform modules — CloudWatch Composite Alarms, X-Ray tracing groups, CloudWatch Logs with subscription filters to OpenSearch, Metric Filters for structured log extraction, SNS alert routing, and Lambda-based alert enrichment
• Dashboard templates — Service-level dashboard (RED metrics: Rate, Errors, Duration), infrastructure dashboard, business metrics dashboard, and SLO tracking dashboard with error budget burn rate
• Alerting playbook — Alert severity definitions, escalation policies, on-call routing rules, and alert fatigue reduction guidelines

Key Architecture Decisions

• OpenTelemetry over vendor-specific agents — Vendor agents lock you into one observability platform. OpenTelemetry provides a vendor-neutral SDK and collector that exports to CloudWatch, X-Ray, Datadog, Grafana, or any OTLP-compatible backend. Switching observability tools becomes a collector configuration change, not an application code change.
• Structured JSON logging over unstructured text — Unstructured logs require regex parsing for every query. Structured JSON logs with consistent fields (timestamp, service, trace_id, level, message, context) enable instant filtering, aggregation, and correlation. CloudWatch Logs Insights queries run 10x faster on structured data.
• SLO-based alerting over threshold-based alerting — Threshold alerts (CPU > 80%) fire for non-impacting events. SLO-based alerts fire when the error budget burn rate indicates you will miss your SLO. A service at 95% CPU but serving all requests within latency targets does not alert. A service at 40% CPU but returning errors that burn error budget does. This reduces alert noise by 70%.
• Composite Alarms over individual metric alarms — Individual alarms for CPU, memory, error rate, and latency create alarm storms during incidents. Composite Alarms combine multiple signals: "Service A error rate > 5% AND latency P99 > 500ms AND downstream dependency health check failing" fires one alarm with full context instead of three separate alarms that an engineer must manually correlate.

Who This Blueprint Is For

• SREs building observability platforms for microservices architectures
• DevOps Engineers replacing ad-hoc monitoring with structured observability
• Engineering Managers trying to reduce MTTR and on-call burden
• Platform teams implementing SLO-based reliability practices

Your First 48 Hours

Deploy the OpenTelemetry Collector as an ECS sidecar using the provided task definition. Configure one service to emit structured JSON logs and traces. Verify that traces appear in X-Ray and logs appear in CloudWatch Logs with trace_id correlation. On day two, create the service-level dashboard using the provided CloudFormation template and configure a composite alarm for the instrumented service. Trigger a synthetic failure (deploy a version that returns 500 errors) and verify the alarm fires with the expected context.

Limitations and Trade-offs

OpenTelemetry adds 2-5% CPU overhead per service for telemetry collection. CloudWatch Logs costs $0.50/GB ingested — high-volume logging (>100GB/day) should use sampling or pre-aggregation at the collector level. X-Ray has a default sampling rate of 1 request per second plus 5% of additional requests; increase this for low-traffic services to get meaningful trace data. CloudWatch dashboards have a limit of 500 metrics per dashboard — complex architectures need multiple dashboards organized by service domain.