System Design Case Study

Real-Time Transaction Reconciliation System for low-latency financial integrity and auditability.

This design focuses on trade ingestion, validation, ledger updates, reconciliation, duplicate detection, and partition-aware processing for financial operations where consistency and audit trails are non-negotiable.

Back to System Design See Relevant Experience

Problem

Continuously reconcile fast-moving transactions without compromising ledger integrity.

A financial platform must ingest trade and transaction events, validate them, update internal ledgers, reconcile against downstream settlement views, and detect duplicates or mismatches quickly enough for operations teams to act.

Requirements

Fast path decisions still need accounting-grade correctness.

80,000 events per minuteDesign assumption for institutional transaction flow.

p95 below 200 msDesign assumption for validation and write admission.

No duplicate ledger impactExactly-once intent for financial state mutation.

End-to-end auditabilityEvery correction and reconciliation decision is traceable.

Mismatch visibilityOperations teams need near-real-time break detection.

High availabilityOutages cannot silently drop financial events.

Architecture Diagram

Trade intake through validation, ledger updates, and reconciliation.

flowchart TB Trade["Trade Sources"] --> Gateway["API Gateway"] Gateway --> Validate["Validation Service"] Validate --> Kafka["Kafka Transaction Topic"] subgraph Processing["Processing Layer"] Ledger["Ledger Writer"] Recon["Reconciliation Engine"] Dedupe["Duplicate Detector"] end Kafka --> Ledger Kafka --> Recon Kafka --> Dedupe subgraph Data["Financial Data Layer"] LedgerDB[("PostgreSQL / Oracle Ledger")] Exceptions[("Exception Store")] Ops["Ops Dashboard"] end Ledger --> LedgerDB Recon --> LedgerDB Recon --> Exceptions Dedupe --> Exceptions Exceptions --> Ops subgraph Recovery["Recovery and Monitoring"] Retry["Retry Topics"] DLQ["Dead-Letter Queue"] Metrics["Metrics / Alerts / Traces"] end Ledger -. retryable failure .-> Retry Recon -. unresolved mismatch .-> Retry Dedupe -. poison event .-> DLQ Validate --> Metrics Recon --> Metrics Ledger --> Metrics

Ingress Flow: Trade Sources -> API Gateway -> Validation -> Kafka Processing Flow: Kafka -> Ledger Writer / Reconciliation / Duplicate Detection Recovery Flow: Exceptions surface to ops while failures isolate into retry and DLQ paths

Cloud-Native Deployment Architecture

How reconciliation services, persistence, and observability run together in production.

flowchart LR Sources["Trade Sources"] --> Ingress["Ingress / Load Balancer"] Ingress --> K8s subgraph K8s["Kubernetes Cluster"] ValidationPods["Validation Pods"] LedgerPods["Ledger Writer Pods"] ReconPods["Reconciliation Pods"] OpsPods["Ops API / Dashboard Pods"] end ValidationPods --> LedgerPods ValidationPods --> ReconPods ReconPods --> OpsPods subgraph Platform["Platform Dependencies"] Kafka[("Apache Kafka")] Oracle[("Oracle / PostgreSQL")] Redis[("Reference Cache")] Prom["Prometheus"] Graf["Grafana"] end ValidationPods --> Kafka LedgerPods --> Kafka ReconPods --> Kafka LedgerPods --> Oracle ReconPods --> Oracle ValidationPods --> Redis LedgerPods --> Prom ReconPods --> Prom Prom --> Graf

Scale Unit: validation, ledger, and reconciliation pods scale by partition and load Persistence: transactional ledger state stays in Oracle / PostgreSQL while Kafka preserves replayability Monitoring: operational breaks, duplicate rates, and write latency flow into dashboards and alerts

Key Decisions

Correctness and operational clarity drive the architecture.

Why Kafka?

Kafka provides durable sequencing, replay support, and ingestion buffering between trade intake and reconciliation workloads.

Why PostgreSQL or Oracle over Cassandra?

Ledger writes and reconciliation references benefit more from transactional guarantees and relational consistency than from wide-column write fan-out.

How are events partitioned?

Partition by account, ledger book, or trade group so ordering-sensitive financial updates stay coherent.

How are duplicates handled?

Use stable transaction identifiers, dedupe tables, and write-side idempotency before any ledger mutation is finalized.

Where is caching useful?

Cache reference data, account metadata, and validation rules, but never treat cache alone as the source of ledger truth.

What happens when a service fails?

Writes pause safely behind queues, retries preserve ordering, and unmatched records surface as reconciliation exceptions instead of disappearing silently.

Trade-Offs

Low latency and strong consistency constantly compete.

Strong consistency

Improves financial correctness, but can reduce raw throughput compared to eventually consistent alternatives.

Partitioned processing

Improves scale and ordering guarantees, but creates hot-partition risk for highly active accounts or instruments.

Real-time reconciliation

Improves operational response, but increases compute cost and pipeline complexity relative to batch-only reconciliation.

Detailed audit logging

Improves compliance and root-cause analysis, but expands storage, retention, and indexing costs.

Scaling And Failure Handling

Financial systems need both elasticity and predictable recovery.

Horizontal scaling

Stateless validation services and consumer groups scale horizontally behind stable partitioning rules.

Retries

Only retry non-mutating or safely idempotent write steps; mutation paths require explicit dedupe controls.

Circuit breakers

Protect downstream ledger dependencies and stop failure cascades from turning transient issues into widespread backlogs.

Dead-letter queues

Malformed or nonreconcilable events are quarantined with full context for operator triage.

Disaster recovery

Combine replicated event logs, database backups, and reconciliation replay workflows to rebuild correct state.

Monitoring

Alert on queue lag, duplicate rates, reconciliation breaks, ledger write latency, and exception aging.

Repository

Implementation status.

This case study is derived from real transaction and reconciliation patterns from my financial systems experience. I have not linked a public repository here because the exact implementation is based on private production-style workflows rather than an open-source codebase.