Real-Time Streaming Architecture Kinesis

The Problem This Blueprint Solves

Your services communicate via synchronous REST calls, creating tight coupling where one slow service cascades latency through the entire request chain. During traffic spikes, downstream services get overwhelmed and return 503 errors that propagate upstream. Adding a new consumer to an existing data flow requires modifying the producer service, deploying both, and coordinating the release. Your architecture cannot absorb load spikes, and adding new integrations is a multi-sprint effort.

This blueprint is the event-driven architecture I built for a logistics platform processing 1.4M daily shipment events across 34 consumer services, where a Black Friday traffic spike of 8x baseline was absorbed without a single failed request or consumer modification.

What You Get

• Architecture diagrams — Event bus topology, producer/consumer patterns, dead letter queue flows, event schema registry, and saga orchestration for distributed transactions (Draw.io)
• Terraform modules — EventBridge custom bus with rules and targets, SQS queues with DLQ configuration, SNS topics for fan-out, Kinesis Data Streams for ordered event processing, and Schema Registry for event contract management
• Event catalog — Template for documenting events with schema definitions, ownership, SLAs, and consumer contracts
• Saga pattern implementation — Step Functions workflow for distributed transactions with compensating actions for each step

Key Architecture Decisions

• EventBridge over SNS for event routing — SNS requires one topic per event type, creating topic sprawl at scale. EventBridge handles hundreds of event types on a single bus with content-based filtering rules. Schema discovery, archive with replay, and native integration with 35+ AWS services make it the superior choice for event-driven architectures beyond simple pub/sub.
• SQS per consumer over shared queue — Each consumer gets its own SQS queue subscribed to the relevant EventBridge rules. Slow consumers do not block fast consumers, each consumer can process at its own rate, and retry/DLQ configuration is per-consumer. Shared queues create contention and make per-consumer error handling impossible.
• Schema Registry for event contracts — Without schema enforcement, producers break consumers by changing event shapes. Schema Registry validates events at publish time, maintains version history, and auto-generates code bindings. Breaking schema changes are caught at deployment, not in production at 3 AM.
• Saga over Two-Phase Commit for distributed transactions — 2PC requires all participants to be available and creates distributed locks that kill throughput. The Saga pattern uses Step Functions to orchestrate a sequence of local transactions with compensating actions (refunds, inventory restocks) if any step fails. Each service manages its own data consistency.

Who This Blueprint Is For

• Backend Architects transitioning from synchronous REST to event-driven communication
• Platform Engineers building shared event infrastructure for multiple product teams
• Engineering leads designing systems that need to absorb unpredictable traffic spikes
• Integration teams adding new consumers to existing data flows without modifying producers

Your First 48 Hours

Deploy the EventBridge custom bus, one SQS consumer queue, and the Schema Registry Terraform modules. Publish a test event using the AWS CLI and verify it arrives in the SQS queue. Attempt to publish an event that violates the registered schema and confirm it is rejected. On day two, add a second SQS consumer queue with a different EventBridge rule pattern. Publish events matching both patterns and verify that each consumer receives only its relevant events. This demonstrates fan-out, content-based routing, and schema enforcement in a working sandbox.

Limitations and Trade-offs

EventBridge has a payload limit of 256KB per event — large payloads must use the "claim check" pattern (store in S3, pass the reference). SQS standard queues provide at-least-once delivery with possible duplicates; consumers must be idempotent. FIFO queues guarantee ordering and exactly-once but limit throughput to 3,000 messages per second per queue. Sagas add complexity — a 5-step saga with compensating actions has 10 possible execution paths to test. Start with 2-3 step sagas and add complexity incrementally.