System Design Case Study
Distributed Order Processing Platform for payment-safe, event-driven order workflows.
This case study shows how I would design a high-throughput order platform spanning API ingestion, payment execution, inventory reservation, logistics handoff, and audit-safe recovery using Kafka and service boundaries.
Problem
Process every order exactly once from user checkout to logistics handoff.
A commerce platform needs to accept bursty checkout traffic, authorize payments, reserve inventory, trigger fulfillment, and maintain an auditable order timeline without blocking the user on every downstream dependency.
Requirements
Throughput, safety, and recovery are all first-class constraints.
Architecture Diagram
Layered order-processing architecture with service zones and event flow.
flowchart TB
Client["Client Applications"] --> LB["Load Balancer"]
LB --> Gateway["API Gateway"]
Gateway --> Order["Order Service"]
subgraph Services["Services"]
Order --> Payment["Payment Service"]
Order --> Inventory["Inventory Service"]
Order --> Logistics["Logistics Service"]
Order --> Notify["Notification Service"]
end
subgraph Messaging["Messaging"]
Kafka["Apache Kafka"]
Retry["Retry Topic"]
DLQ["Dead-Letter Queue"]
end
Order --> Kafka
Kafka --> Payment
Kafka --> Inventory
Kafka --> Logistics
Kafka --> Notify
Payment -. failure .-> Retry
Inventory -. failure .-> Retry
Logistics -. poison event .-> DLQ
Notify -. poison event .-> DLQ
subgraph Data["Data"]
OrdersDB[("PostgreSQL Orders")]
Redis[("Redis Cache")]
Audit[("Audit Store")]
end
Order --> OrdersDB
Order --> Redis
Order --> Audit
Payment --> OrdersDB
Inventory --> OrdersDB
Logistics --> OrdersDB
Notify --> Audit
subgraph Observability["Observability"]
ELK["ELK Stack"]
Prom["Prometheus"]
Graf["Grafana"]
end
Order --> ELK
Payment --> Prom
Inventory --> Prom
Logistics --> ELK
Prom --> Graf
Request Flow: Client -> Gateway -> Order Service
Event Flow: Order Service -> Kafka -> Payment / Inventory / Logistics / Notification
Recovery Flow: Failed consumers -> Retry Topic -> DLQ -> Operator Replay
Cloud-Native Deployment Architecture
How the platform runs across Kubernetes, Kafka, data stores, and monitoring.
flowchart LR
Ingress["Ingress / Load Balancer"] --> K8s
subgraph K8s["Kubernetes Cluster"]
GatewayPods["Gateway Pods"]
OrderPods["Order Service Pods"]
ConsumerPods["Consumer Pods"]
WorkerPods["Worker Pods"]
end
GatewayPods --> OrderPods
OrderPods --> ConsumerPods
ConsumerPods --> WorkerPods
subgraph Platform["Platform Dependencies"]
Kafka[("Apache Kafka")]
Schema[("Schema Registry")]
RDS[("AWS RDS")]
Redis[("Redis")]
Prom["Prometheus"]
Graf["Grafana"]
end
OrderPods --> Kafka
ConsumerPods --> Kafka
Kafka --> Schema
OrderPods --> RDS
ConsumerPods --> RDS
OrderPods --> Redis
WorkerPods --> Prom
ConsumerPods --> Prom
Prom --> Graf
Scale Unit: stateless pods scale horizontally behind the ingress layer
Persistence: Kafka retains event history while RDS stores transactional order state
Monitoring: metrics, logs, and lag dashboards drive incident response and recovery
Key Decisions
Every major component exists to protect throughput and correctness.
Why Kafka?
Kafka decouples order intake from downstream payment, inventory, and logistics services while preserving replayability and durable audit trails.
Why PostgreSQL?
Order state transitions and payment references benefit from strong consistency, relational modeling, and transactional writes more than wide-column throughput.
Why Redis?
Idempotency keys, recent order locks, and short-lived deduplication checks fit well in an in-memory cache with aggressive TTL management.
How are events partitioned?
Partition by `orderId` so each order’s lifecycle stays ordered while still distributing work horizontally across consumers.
How are duplicates handled?
API requests carry idempotency keys, consumers persist processed event tokens, and payment requests use provider-safe idempotent references.
What if a service fails?
Failed consumers back off, publish to retry topics, and eventually route poison events to a dead-letter topic for operator review.
Trade-Offs
Decoupling helps scale, but it changes how consistency is achieved.
Kafka
Improves decoupling and replayability, but introduces eventual consistency, broker operations, and partition management complexity.
Saga orchestration
Reduces cross-service synchronous coupling, but requires careful compensation design and observability to understand incomplete flows.
PostgreSQL over Cassandra
Improves transactional integrity for payments and orders, but sacrifices some write scalability that a wider distributed store could offer.
Caching
Redis improves speed for dedupe checks and hot order lookups, but adds cache invalidation and consistency considerations.
Scaling And Failure Handling
Horizontal scale only matters when failure paths are equally designed.
Horizontal scaling
Gateway and stateless services scale behind load balancers; consumer groups scale with Kafka partitions.
Partition strategy
Use `orderId` for event ordering and shard high-cardinality hot keys with tenant-aware partitioning when needed.
Retries and circuit breakers
Transient failures use exponential backoff, while circuit breakers protect order intake from unstable downstream services.
Dead-letter handling
Unrecoverable events land in DLQ topics with replay tooling and operator dashboards for targeted reprocessing.
Disaster recovery
Replicate Kafka and databases across zones, snapshot order state, and retain event history for controlled restoration.
Monitoring
Track topic lag, payment failures, idempotency collisions, inventory reservation misses, and order-state aging.
Repository
Public implementation reference.
Public reference implementation: jeshwinwilliam/distributed-event-processing-system. The repository reflects the core distributed-event, retry, dead-letter, and consumer-scaling patterns behind this architecture.