5 minute read

At the end of part 3 I wrote: “the live demo is down now, due to AWS costs.” That sentence bothered me for days after publishing. Writing “I built this” in a portfolio with nothing to click is a completely different thing from clicking a link and watching an order book actually move.

So I put it back up — with a completely different layout. It’s running at exchange.agentmichael.me.

What I removed

The original setup was 15 services on ECS Fargate, plus RDS MySQL, ElastiCache Redis, two MSK Kafka brokers, two ALBs, and a NAT gateway. Textbook. And it bills an uncomfortable amount every month for something one person keeps up as a demo.

Most of that cost comes not from traffic but from the mere fact of being switched on. MSK brokers bill by the hour whether or not anyone places an order, and so does the NAT gateway. A handful of people hit the demo per day, while the availability design was built for a real service.

So I stripped out every managed service and packed the whole thing into containers on a single box.

  • One EC2 r7g.large (Graviton, arm64)
  • 21 containers — the Rust matching engine, 11 Go services, 3 Next.js apps (trading, admin, agent), MySQL / Redis / MongoDB / Kafka / ZooKeeper, and Caddy out front
  • Three CloudFront distributions in front. The domains and certificates were already attached, so I swapped the origins instead of creating new ones

Availability clearly went down. If this one box dies, everything dies. But what a demo needs isn’t multi-AZ redundancy — it’s being there when someone clicks. Deciding what level you actually need and building to that is, I think, what design means.

Where do you build the images?

This was the first wall. The EC2 box has 2 vCPUs, and building 3 frontends + 11 Go services + the Rust engine there takes over an hour.

Luckily my Mac is arm64 and r7g is Graviton (arm64) too. Same architecture means I can build locally and ship the images as-is. I didn’t even create an ECR repository — there’s no reason to stand up a registry for a one-box deploy, so I just pipe everything over in one go.

docker save "${IMAGES[@]}" | gzip -1 | ssh ec2-user@$HOST 'gunzip | docker load'

Opening a fresh ssh connection per image piles up round trips, so it all goes through a single pipe.

Put a frontend and an API on one domain and the paths collide

Locally this never came up: the frontend was on port 3000 and the gateway on 8080. The moment they share a domain, they collide.

Next.js has an /exchange/BTC_USDT page, and the gateway also considers /exchange/ its own. Two things claiming the same address.

I settled it by putting an /api prefix on the API side and stripping it in Caddy before forwarding.

handle /api/* {
    uri strip_prefix /api
    reverse_proxy gateway:8080
}

The frontends bake https://exchange.agentmichael.me/api in as their API base at build time. From the gateway’s point of view the original path arrives with the prefix already gone, so not a single line of its code changed.

Telling three CloudFront distributions apart when they share one origin

The trading, admin, and agent apps each have their own CloudFront distribution, but the origin is one EC2 box. Caddy has no way to know which app an incoming request is for.

I solved it by giving each app its own origin hostname. Three Route53 records — o-exchange / o-admin / o-agent — all pointing at the same box. Which name the connection was made under is visible while the TLS connection is being established, so Caddy just splits on that.

I stared at a 502 for a while here. CloudFront has a setting for which headers it forwards to the origin, and I first picked “forward everything the viewer sent.” That forwards the Host header as the viewer’s value (exchange.agentmichael.me) too. My Caddy config only has an o-exchange block, so nothing matches.

What made the symptom confusing is that the certificate was fine. The connection is established under the name o-exchange, so TLS passes, and then the HTTP stage has nowhere to go. Switching to the policy that keeps the Host header as the origin’s and forwards everything else fixed it immediately.

The deploy reported success and not a single trade could go through

The longest hunt wasn’t infrastructure — it was the schema.

I applied the initial schema files and the numbered migration files in one pass with ls | sort. That pushes the base schema, named schema.sql, behind files like 043_.... The numbered migrations then tried to alter tables that didn’t exist yet and failed one after another, and the deploy script reported green with core tables like exchange_trade missing entirely.

Every service came up. The pages rendered. But no order would go through, and finding out why took a long time.

I fixed two things. Application order is now stated as an explicit dependency order (initial schema → base schema → numbered migrations) rather than a name sort, and a final check verifies the core tables actually exist, halting the deploy if they don’t.

for t in exchange_trade exchange_order exchange_coin robot_params member; do
  ... exit 1 if missing
done

In effect I changed the condition for a green deploy from “the commands finished without errors” to “the things the service can’t exist without are present.” A deploy that fails quietly is far more dangerous than one that fails loudly.

Things I ran into along the way

Pipe a script into ssh and the back half disappears. I fed the remote script in through ssh’s stdin, and a docker compose exec -T inside the script consumed the same stdin — swallowing everything after it. It ran as far as the MySQL readiness check, and the schema step never executed at all. Sending the script as a file and then running it fixed it.

Never write $var:latest in zsh. The :l is parsed as a lowercase-conversion modifier, so agent-web:latest becomes agent-webatest. It caught me once, and docker load’s “invalid reference format” tells you nothing about why.

What’s left

The trading UI, the admin back office, and the agent portal all run on this setup. Prices update in real time, and the market-making bots fill the book.

In part 3 I said I’d learned everything I wanted from building it, so I had no regrets about taking it down. Having put it back, I learned about as much from shrinking the deployment as from building it. On ECS Fargate the managed services hid a set of decisions — when and in what order the schema gets applied, how the front layer tells the back layer apart. This time I had to make all of them myself.

There’s plenty of material on designing things to scale up. Designing them down seems to be something you learn by running into it.

Updated: