In-depth architectural analyses, systems design deep-dives, and performance engineering notes written during ongoing study and system modeling.
#SystemsDesign#EventSourcing#WAL#Scala3
From 0 to Scale
A simple example of scaling an event sourcing system. Deconstructing an order service from single-node transactional CRUD to high-throughput Write-Ahead Logging (WAL), in-memory projections, and group commit based on anpaulin/order-api.
Straightforward, simple, and concise. But even in this basic system, subtle race conditions and scaling bottlenecks emerge as soon as concurrent clients interact with the service.
2. Network & In-Flight Timing: The Commit Boundary
Even in a basic single-server setup, network timing anomalies happen because a write is never a single instantaneous moment.
A write operation involves multiple steps wrapped inside a database transaction: parsing the payload, validating business rules, acquiring connection sockets, executing SQL INSERT/UPDATE, writing to disk, and finally issuing a COMMIT.
Because of this transaction window, if a client fires a Write (Create) and immediately fires a Read (Query), the fast read query can arrive, execute, and return before the write transaction has fully committed, even if the write was dispatched first!
Figure 2: The In-Flight Race Window: Reads only see data once a transaction successfully COMMITS.
⚠️ It's All About the COMMIT Boundary
Under standard relational isolation (READ COMMITTED), writes remain invisible to other queries until the final COMMIT succeeds. Only after a write is fully completed does the database guarantee that subsequent reads return the most up-to-date state.
Beyond network timing, a separate and deeper challenge lies in database schema and transactional concurrency design.
💡 The Reality: For 95% of Applications, A Good Relational Schema is All You Need
Before exploring advanced scaling, it is worth stating plainly: for the vast majority of systems, a single ACID-compliant relational database with a normalized schema will completely suffice.
Instead of forcing all state into a single mutable row, you simply normalize the domain into separate tables:
A standard database transaction wraps the operation:
BEGIN;
INSERT INTO refunds (order_id, refund_amount) VALUES (123, 100.00);
UPDATE orders SET status = 'REFUNDED' WHERE id = 123;
COMMIT;
The UNIQUE constraint on refunds(order_id) physically prevents double-refunds at the database engine level, rollbacks are built-in, and you get complete auditability for free.
The Single-Row Mutation Dilemma
Concurrency traps emerge when systems store all operational state inside a single mutable row (e.g. updating status and price on the same row):
CREATE TABLE orders (
id BIGSERIAL PRIMARY KEY,
status VARCHAR(32) NOT NULL DEFAULT 'ACTIVE', -- ACTIVE, REFUNDED, CANCELLED
price NUMERIC(10, 2) NOT NULL,
currency VARCHAR(3) NOT NULL DEFAULT 'USD',
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);
💡 Effective Database Design vs. Scaling Concerns
It is vital to understand the relationship between database design and scaling:
Schema-Level Invariants: Use exact financial precision (e.g. NUMERIC/DECIMAL over floating-point FLOAT) and explicit constraints (e.g. CHECK (price >= 0)) to guarantee baseline integrity at rest.
Scaling Without Correctness is Just Fast Corruption: High throughput is meaningless if concurrent requests enter an invalid state (e.g. charging refunded orders or double-spending). While some domains can tolerate eventual consistency (such as view counts or likes), transactional systems (orders, billing, inventory) must always remain accurate and valid without exception.
The Core Challenge: Many common scaling techniques (read replicas, caches, async message queues, and sharding) introduce replication lag and out-of-order anomalies that actively threaten correctness unless your database and concurrency architecture explicitly protects against them.
The Concurrent Mutation Race
Assume a situation where two simultaneous changes are placed to our system for a single order by two different departments:
Customer Service: Tries to refund the order (transitioning status to REFUNDED).
Billing Department: Tries to adjust the order price (e.g. applying a retroactive fee or discount).
Our Basic Business Rule: We shouldn't change the price of orders that are already refunded (we only update the price if the order is currently ACTIVE).
In the simplest implementation, our application performs a READ and a WRITE operation separately: first querying the database to fetch the row (SELECT), evaluating the business rule in application memory (if (status == "ACTIVE")), and then issuing a separate database write (UPDATE).
Because the read and the write are two separate operations with a gap in between, there are 4 possible execution scenarios when two departments act at the same time:
CS finishes completely first: CS reads order as ACTIVE ➔ marks it as REFUNDED. Billing then reads order as REFUNDED, fails condition check, and makes no changes.
✓ Correct outcome: Order is refunded at original price; Billing is safely rejected.
Billing finishes completely first: Billing reads order as ACTIVE ➔ updates the price to $120. CS then reads order as ACTIVE (at the new price) ➔ marks it as REFUNDED.
* Order refunded with new price: While we didn't strictly break our database rule on paper, from a business perspective it still looks like we did (customer gets refunded an altered amount, NOT GOOD).
Interleaved (CS WRITES before BILLING WRITES): CS reads order as ACTIVE. Billing reads order as ACTIVE. CS writes and marks order as REFUNDED. Billing then writes and updates the price based on its stale read.
🚨 Direct rule violation: Billing updated the price of an already-refunded order!
Interleaved (BILLING WRITES before CS WRITES): Billing reads order as ACTIVE. CS reads order as ACTIVE (seeing original $100 price). Billing writes and updates price to $120. CS then writes and marks order as REFUNDED.
* CS authorized $100, DB recorded $120: While the SQL write technically passed the ACTIVE check, the customer gets refunded money CS never authorized (NOT GOOD).
* Key Takeaway: In Scenarios 2 and 4, even though we technically didn't break our database condition check at the exact millisecond of the write, it still looks like we broke our business rules to both the customer and finance team.
How Do We Design the DB to Prevent This?
To prevent concurrent lost updates and protect our business rules on mutable rows, we can explore four architectural patterns, ordered from heaviest hammer to the cleanest standard:
Option 1: The Database Hammer: Serializable Isolation Level
Wrap the read and update inside a transaction with the strictest database isolation level:
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
BEGIN;
SELECT status, price FROM orders WHERE id = :id;
-- Perform check in application code...
UPDATE orders SET price = :new_price WHERE id = :id;
COMMIT;
How it works: The database engine automatically tracks what data transactions read. If two transactions concurrently read and try to modify the same order, the database aborts the second one with ERROR: 40001 serialization_failure.
Trade-off: Heavy performance overhead, high transaction abort rates under load, and requires complex application retry loops.
Option 2: Manual Row Locking: Pessimistic Locking (SELECT FOR UPDATE)
Screw it, let's just lock the entire row manually from the start, read, perform our check, update, and commit:
BEGIN;
SELECT * FROM orders WHERE id = :id FOR UPDATE;
-- Application checks: if (order.status != "ACTIVE") ROLLBACK;
UPDATE orders SET price = :new_price WHERE id = :id;
COMMIT;
How it works: As soon as the first transaction executes SELECT ... FOR UPDATE, the database locks the row. The second transaction is physically blocked from reading or writing until the first transaction commits.
Trade-off: Eliminates races, but holds database connections open, serializes all traffic on that row, and drastically reduces throughput.
Option 3: Atomic State Guard (Fusing Check & Write into the UPDATE)
Instead of a separate SELECT and subsequent UPDATE, fuse the check and write directly into a single SQL statement:
UPDATE orders
SET price = :new_price
WHERE id = :id AND status = 'ACTIVE';
How it works: The database automatically acquires an exclusive row lock for the single statement. If CS is updating first, Billing's update blocks and waits. Once CS commits (status becomes REFUNDED), Billing wakes up, re-evaluates status = 'ACTIVE' against the new row, and modifies 0 rows (rejected with 409 Conflict).
💡 HINT: The Remaining Flaw
If Billing acquires the lock before Customer Service, the price still silently changes to $120 right before the refund commits. We solve this semantic dilemma with Event Sourcing & Ledger-style transactions (covered in Section 3 below)!
Option 4: The Clean Standard: Optimistic Concurrency Control (OCC with Versioning)
Add a monotonically increasing version column to the table. When updating, include the version you read in the WHERE clause:
UPDATE orders
SET price = :new_price, version = version + 1
WHERE id = :id AND version = :expected_version;
Why this is so clean: We don't hold any heavy locks. If either department commits their update first, the version increments (v1 ➔ v2). The second update fails (0 rows updated) no matter which department sent it! The application detects 0 rows and returns 409 Conflict, guaranteeing we know that something changed in this mutable row before proceeding.
⚠️ The Inherent Limitation of Mutable Rows
While OCC successfully stops Billing from silently updating the price to $120 behind Customer Service's back (because CS's initial v1 update fails once Billing bumps it to v2), Customer Service still has the option of pushing another update. If they simply refresh and click refund again, we end up right back in the same bad state: a single mutable row that says we refunded $120!
3. The Scaling Dilemma & The CAP Theorem
A single monolithic server and a single relational database are simple, concise, and easy to reason about. But what happens when our traffic grows and we need to scale? A single server can only handle so many concurrent requests before hitting CPU, memory, and disk I/O limits.
As soon as we move beyond a single machine (whether by adding read replicas, partitioning databases, or separating write logs from read caches), we run directly into the fundamental law of distributed systems: Brewer's CAP Theorem.
Consistency (C)Every read receives the latest write (or fails). All clients see the exact same data at the same moment.
Availability (A)Every non-failing node returns a valid response. The system never crashes or errors out, even if the data returned is slightly stale.
Partition Tolerance (P)The system survives network breaks. It continues to operate even when communication links between servers drop, lag, or fail.
⚖️ The CAP Trade-Off: You Must Choose Between CP and AP
In the real physical world, Network Partitions (P) are unavoidable: cables get cut, switches drop packets, and garbage collection pauses mimic network failures. You cannot "choose" CA.
Therefore, whenever a network partition occurs, you must make an architectural choice:
CP (Consistency + Partition Tolerance): If nodes cannot communicate to verify the latest state, the system rejects or delays the request to avoid returning corrupted or stale data (prioritizing safety over availability).
AP (Availability + Partition Tolerance): The system responds immediately with whatever local state it has, accepting that data might be temporarily out-of-sync (Eventual Consistency).
📈 When Demands Outweigh Single-Instance Capacity: The Read Explosion
Imagine our application traffic surges: instead of a few queries a second, 25,000 active users are concurrently refreshing order pages, tracking shipments, and checking purchase histories.
Even though actual order writes (creations/payments) remain moderate, the massive flood of read queries exhausts the database connection pool, pegs CPU cores at 100%, and starves disk I/O. Both reads and writes begin timing out.
In web applications, reads typically outnumber writes by 10:1 or more. We solve this first scaling bottleneck by separating reads from writes using a Primary-Replica (Master-Slave) architecture:
Writes go strictly to Master (Primary): All INSERT, UPDATE, and DELETE operations are sent exclusively to a single Master node.
Reads go to Slaves (Replicas): All SELECT queries are distributed and load-balanced across multiple Read-Only Slave nodes.
Replication Stream: The Master continuously streams its write log (binary log / WAL) to keep all Slaves up to date.
Figure 3: Primary-Replica (Master-Slave) Architecture Separating Read and Write Paths
The Replication Trade-Off: Synchronous vs. Asynchronous
How and when data propagates from Master to Slaves creates a direct application of the CAP theorem:
Option A: Synchronous Writes
Master waits for all Slaves to acknowledge the write before committing and replying 200 OK to the client.
Good: Better consistency across all nodes; zero data loss if Master fails.
Bad: High write latency (waits on slowest Slave); writes stall completely if one Slave hangs.
* Physical World Catch: While consistency is much better, writes to multiple slaves never land at the exact same physical millisecond. Reads hitting different slaves during the write window can still briefly return out-of-sync data.
Option B: Asynchronous Writes (Standard)
Master commits immediately and replies to client. Replicas receive updates in the background.
Good: Maximum write throughput; ultra-low write latency.
Bad:Replication Lag: Reads hitting Slaves will temporarily see stale data; failover risks losing un-replicated writes.
💥 The Breaking Point of Master-Slave: When Writes Explode
While Master-Slave successfully scales reads, it has a fatal architectural limitation: all writes are still forced through a single Master node.
Two real-world situations inevitably break this architecture:
1. High-Volume Write Surges (Flash Sales / Black Friday / IoT): When 20,000 users click "Buy Now" simultaneously, every write hits the single Master. Relational writes perform heavy random disk I/O (updating table pages, updating B-Tree indexes, acquiring row locks, and flushing undo logs). The Master's disk IOPS saturates and write transactions grind to a halt.
2. Microservices & Sharding Break Single-DB ACID: When data grows so large that we shard across multiple databases or split into microservices (OrderService, PaymentService, InventoryService), we can no longer run a single multi-table ACID transaction. Distributed Two-Phase Commits (2PC) introduce massive network latency and fail completely when any node is unreachable (CAP theorem).
This is the exact threshold where we move to CQRS (Command Query Responsibility Segregation) and Event Sourcing:
Command (Write) Side: Writes bypass heavy relational index rewrites and table locks. Instead, they are appended as lightweight, immutable events to a sequential Write-Ahead Log (WAL) at tens of thousands of writes per second.
Query (Read) Side: Independent read projections consume the event log asynchronously to build fast, denormalized read views tailored specifically for UI queries with zero lock contention on the write path.
💥 Operational Disaster: The "Accounting Incident" (Mutable Overwrite Nightmare)
Beyond write throughput, mutable databases create severe operational risks. Imagine a bad actor (or a buggy script) in Accounting runs an unauthorized SQL script that overwrites prices across 5,000 orders in our database:
Destructive Overwrites: The previous prices were permanently erased the moment the UPDATE executed. The database only knows the corrupted data in front of it.
The Painful Rollback: We might have a database backup from 2 hours ago. But restoring that backup rolls back everything, destroying legitimate orders, customer sign-ups, and payments that occurred over the last 2 hours.
Manual Triage: Engineering is forced to dig through cryptic, unstructured server logs, guess which changes were missed, manually reconcile balances, and notify customers that account data was stale or corrupted.
🏦 Real-World Aside: How Traditional Banks Avoid This on ACID DBs
Ever wonder how traditional banks avoid this disaster while still using classic ACID relational databases?
Banks turn their ACID database into an append-only ledger. They primarily only ever perform INSERT statements. They almost never issue UPDATE or DELETE statements on ledger tables.
If you deposit $100, they INSERT a deposit record. If a teller makes a mistake and deposits $1,000 instead of $100, the bank does not overwrite the existing row. Instead, they INSERT a compensatory adjustment record (e.g. -$900 Reversal).
Your current account balance is never a raw mutable cell in a row; it is computed by folding over all historical append-only entries:
Current Balance = SUM(Credit Entries) - SUM(Debit Entries)
The Paradigm Shift: From Mutable Rows to Immutable Events
What if, instead of mutating a single row over and over and praying that state never gets corrupted, every change is treated as an immutable "Event"?
By modeling our system around an append-only sequence of events, we unlock massive architectural superpowers:
Superpower
What It Gives Us
1. Complete First-Class Audit Trail
We maintain a full, tamper-evident history of what operations happened, when, and by whom (without digging through messy binary logs).
2. Deterministic State Rebuilding
Because current state is simply a projection calculated from past events, we can replay events up to any timestamp, isolate bad events, and rebuild the exact system state deterministically.
3. Event Listeners & Decoupled Pub/Sub
We can push events to different listeners across our ecosystem: one listener sends customer confirmation emails, another notifies shipping, and another updates search indexes.
4. Cheap Long-Term Storage
Append-only logs can be streamed and archived to cheap cold storage (like S3 or GCS) indefinitely, ensuring we always know the absolute history of how we arrived at our current state.
The Order API WAL Architecture
As implemented in the order-api microservice, write requests append to a high-throughput Write-Ahead Log (WAL) before projecting into an in-memory cache for zero-latency queries:
Zero State Drift: State projections are never updated in memory until the event is safely committed to the append-only log. If the server crashes or the disk fills up, the in-memory cache remains consistent upon replay on startup.
* The Atomicity Rule: Appending to the WAL audit log and updating the in-memory state must happen synchronously and atomically per entity (they either both happen or neither does). If an in-flight gap allows concurrent requests to interleave between appending to the log and updating memory, the system will read stale state, record duplicate or conflicting events in the log, and cause memory-disk drift.
Why this still easily beats a classic RDBMS: Even with synchronous ordering enforced, our operations remain lightning fast. Appending sequentially to the end of a file bypasses heavy B-tree index traversals, random disk I/O, and table-lock contention, while updating a RAM data structure takes nanoseconds.
6. Overcoming Performance Hurdles (The 250 req/s Ceiling)
Implementing strict synchronous persistence on each request introduces OS-level bottlenecks. In order-api, the performance evolution followed distinct phases:
Phase & Bottleneck
Root Cause
Architectural Remedy
1. Netty Thread Starvation
Blocking file I/O executed on Play's default CPU execution context.
