Building an Exchange with Claude Code (5) — Where the Money Quietly Leaks

The scariest bugs in building an exchange were never the ones that crashed the server. A crash pages you, and then you fix it.

The scary ones were the bugs that were wrong quietly, without raising a single error. Every service green, every page rendering fine — and somewhere, a user’s money frozen. No amount of log reading turns up an error, because there isn’t one. I found about five of these. This is that record.

Diagrams make this easier to follow, so I put three of them — deployment, trade flow, market data pipeline — in the case study on my portfolio. They’re drawn from the same code this post is about.

1. The order book lives only in memory

The matching engine’s order book is two BTreeMaps. Nothing is written to disk. That’s why it’s fast, and it’s also the problem.

When the engine dies, the book vanishes entirely. But the orders in the database stay marked “open.” The engine never reads the order table, so those orders become ghosts belonging to no book. They can’t fill, and cancelling does nothing because the engine has never heard of them. The frozen funds never come back.

I measured how bad it was. Eighty minutes after a restart, 94–98% of open orders predated the restart. Nearly everything a user saw as “open” was already dead.

The fix is simple. On startup, read each symbol’s open limit orders in time order and put them back into the book. Market orders are never resting in the book, so they aren’t candidates.

One decision needed making: should restoration run matching as it goes? I decided no. Restoring means returning to the state just before the crash, not catching up on fills that didn’t happen. Batch-matching on startup would spike the price at that instant — and that’s a fill the user never agreed to when they placed the order.

There’s a cap, too. But instead of silently dropping the overflow, it logs the count. Hitting the cap is itself the signal that something needs cleaning up.

2. One transaction per trade never catches up

Settling a single trade costs: one dedupe INSERT, two order reads and two writes, up to four wallet updates, and a commit.

By far the most expensive part is the commit’s disk sync — and that cost scales directly with the number of commits.

I measured it. About 30 trades/sec per Kafka partition. Meanwhile the market-making bots produced 39/sec. Nine trades fall behind every second. The gap has no reason to close, so it widened continuously, and user orders queued behind them landed hours later.

What’s interesting is that this never surfaces as an error. Processing is working correctly. It’s just slow. Without a distinct “lag” metric in monitoring, you never find out.

I switched to batching. Commits now scale with batches instead of trades, and multiple fills against the same order collapse into one update.

Why batch by symbol is the crux. The Kafka partition key is the symbol and the consumer runs one goroutine per partition, so a given symbol’s messages are always handled in order by exactly one goroutine. Batch by symbol and only one goroutine ever touches a batch, so ordering can’t invert. Choosing the partition key so that it doubles as the ordering-guarantee unit — not just for spreading load — pays off precisely here.

Batches cap at 200 trades or 200ms. Bigger batches win more per commit, but they also lock more order rows per transaction and cost more to roll back on failure. And a thinly traded symbol’s fill must not sit waiting for a batch to fill up, hence the time bound.

3. Offsets advance on “read,” not on “processed”

This one took the longest to find.

The consumer runs with auto-commit. Offsets advance on a 5-second timer, and the criterion is not “did processing finish” but “did ReadMessage pull it.”

A pulled message isn’t processed immediately. It goes into a per-partition lane queue (max 256), and when it comes out of there it lands in the settlement batch described above. If the process dies while a message sits in either stage — SIGKILL, OOM, container replacement during a deploy — that trade never arrives. The offset has already moved past it.

In money terms: the engine filled it, the counterparty got settled, but this side’s order stays open and the funds stay frozen. And nothing errors.

I measured the leak. Over two hours, 8 of 214,482 trades had vanished — 0.004%. And the timestamps clustered exactly at that day’s container restart. That’s the moment the cause clicked.

0.004% sounds small. But this isn’t a probability — it’s something that happens every single restart. And each of those 8 is somebody’s frozen money.

4. Recovery — the difference between two consumers

The textbook fix is committing offsets manually after processing. That closes the window itself.

But something usable was already there. The market service consumes the same trade topic into a separate table. Different consumer group, so its offsets move independently. The odds of both losing the same message are low.

Which gives you:

present in market’s table, absent from settlement = what settlement dropped

Found ones get replayed through the normal settlement path. A dedupe key means re-inserting an already-settled trade doesn’t double-apply. That means the detection can be a little loose and still be safe — misjudging a non-lost trade as lost costs nothing. When you have that property, you can build recovery machinery far more aggressively.

The limitation, stated honestly: if market loses the same message too, there’s no trace anywhere and it can’t be recovered. What this does is not close the window, but fill it in on its own when it opens.

And every single recovered trade gets logged. Recovering something means a loss occurred. A recovery mechanism that runs quietly and well makes the problem look like it’s gone, which is the most dangerous state of all.

5. You can’t leave money frozen just because you haven’t found the cause

Some orders reach their full quantity but stay marked “open.” The engine doesn’t have them. The UI keeps showing them as open, and cancelling produces no confirmation because the engine doesn’t know them. The frozen funds never release.

I found one cause. When a fill arrived late for an order already closed by cancellation, settlement resurrected it back to “open.” A guard fixed that.

But a small number kept appearing afterward. I still haven’t found the remaining cause.

That required a judgment call. Not having found the cause is no reason to leave a user’s money locked up. So there’s a separate sweeper.

You can only ship something like that if you can articulate why it’s safe. It only targets orders whose filled quantity already equals the order quantity. The engine cannot fill those any further — there’s nothing left. So closing them here can’t lose a late-arriving fill.

Plus a grace period. A just-filled order may still be sitting inside a settlement batch. Cutting in during that window means we close it first and settlement then discards the fill as belonging to a finished order — manufacturing a problem that didn’t exist.

This one logs every sweep too. Having something to sweep means the bug upstream is still there.

6. Why chart volume was uniformly too low

This one doesn’t leak money, but it’s the same species of quiet error, so it belongs here.

One-minute candles are aggregated every 30 seconds. But the aggregation start point wasn’t floored to the minute boundary.

Run at 09:25:37 and the start becomes 09:15:37. So the 09:15 candle gets a value summed only from second 37 onward — and that value overwrites the correct one via ON DUPLICATE KEY UPDATE.

Running every 30 seconds meant the boundary kept sweeping forward, doing this to every minute it passed. Measured, stored volume was 0.9–17% of actual, and the back-computed offset was 50–59 seconds, so each candle retained only its last 1–10 seconds. Fifteen-minute and hourly candles sum these, so chart volume came out uniformly small.

The symptom was confusing for a reason. One-minute candles have a read path that recomputes recent intervals from trades, so only the last few minutes were correct and everything before was wrong. Open the screen and the present looks fine.

The same class of mistake was on the read side. If the requested start lands mid-candle, the first candle goes out half-filled. Its open and volume read lower than reality, but the chart shows it as a complete candle. Flooring to the boundary fixed it.

In code that handles time, boundary handling goes wrong in almost exactly this way. And almost always quietly.

7. What the matching engine actually looks like

I’ve talked only about consistency, so here’s the engine itself, briefly.

The order book expresses price-time priority directly in its data structures.

• Buys: BTreeMap<Reverse<Decimal>, VecDeque<Order>> — higher price first
• Sells: BTreeMap<Decimal, VecDeque<Order>> — lower price first
• Within a price level, VecDeque gives first-come order

All money is rust_decimal. Compute money in floating point and you will eventually have an incident.

Concurrency is one dedicated OS thread per symbol — not a tokio task. Orders reach that thread through a bounded channel of 4096. This means the book needs no lock, since exactly one thread touches it, and ordering within a symbol cannot invert.

Two details I enjoyed.

Market order remainders. A market order’s unfilled remainder isn’t rested in the book — it’s cancelled. But that cancellation must be reported back to the caller. Without it the DB order stays “open” forever and the frozen funds stay locked. It’s the exact same shape as the ghost-order bug above.

Self-trade prevention with an exception. You have to stop one person’s buy and sell from matching each other — but generating volume and candles through self-trades is precisely the market-making bot’s job. So orders carry their origin, and bots are exempt. The default is “customer,” so an order missing its origin falls on the safe side.

The thread running through all of it

Look at the six together and they’re the same shape.

Being quietly wrong is more dangerous than loudly dying. Changing the deploy script’s green-light condition from “the commands finished without errors” to “the things the service can’t exist without are present” came from the same place.

A recovery mechanism must be its own alarm. If recovered trades aren’t zero, if swept orders aren’t zero, the bug upstream is still there. A recovery job that hums along quietly hides the problem.

Only automate a fix when you can say out loud why it’s safe. “Only orders that can’t be filled further,” “a dedupe key makes double-insertion harmless” — with reasons like that you can automate aggressively. Without them, a human should look.

Write down what you haven’t figured out, too. The remaining cause in section 5 is still unknown. Better to record that you don’t know and contain the symptom than to pretend otherwise. The next person reading it is usually you, six months later.

---

That’s the part of building an exchange that I expect to outlast the code. How to write an order book in Rust is a search away; losing 8 trades because you didn’t know when offsets advance was not.

You can watch it run at exchange.agentmichael.me. Earlier posts: part 3 — the Go+Rust re-platform, part 4 — putting it back on one EC2 box.