The things I keep telling teams. Log the unhappy paths, stop trusting averages, decide how things fail before they fail, and don’t add infrastructure you haven’t measured a need for.
Every now and then someone pulls me into a conversation about how their team should build a service. I’m on the platform team at AmaliTech, so I guess that comes with the job. And honestly, whatever the question is, my answer usually ends up being some version of the same thing: use the boring option.
I mean that. Good system design is rarely about clever tricks. Most of the time it’s taking things that have already been tested to death and putting them where they belong. So in this post I want to talk about two things I keep seeing teams treat as afterthoughts, logging and failure handling. Then I’ll finish with a recent one where someone made a perfectly reasonable proposal and I still argued against it.
Here’s the question I keep coming back to: how do you find out something is broken before a customer tells you?
For me it starts with logging, and a lot of it, on the unhappy paths. Say a function runs a bunch of checks to decide whether an endpoint should return a 422. Log which check failed. In billing code, I’d log every single decision, even the ones where nothing happens. Something like “not billing this event because the account is on a trial plan.”
Yes, I know. It clutters the code, and your functions won’t look as elegant. Do it anyway. One day a major client is going to show up with a stream of 422s, and you’ll have to tell them exactly what they sent wrong. You can only do that if the code wrote it down.
The third check fails, so the request stops there with a 422. The log line says which check failed and what the client actually sent. Check names and values are illustrative.
Then there are the operational basics. You want CPU and memory on every host or container, how deep your queues are, and how long each request or job takes. And for anything user-facing, please don’t trust averages. Look at p95 and p99 latency instead, because those tell you how slow the slowest 5% and 1% of your requests really are. A handful of very slow requests should actually worry you, because they tend to come from your biggest, most important users. Your average can look perfectly healthy while a few accounts find the product unusable.
The mean (210 ms) sits in the healthy zone. p95 and p99 show what the slowest requests actually feel like. Illustrative data.
Spend real time thinking about what your system does when things go badly wrong. One cheap bit of insurance is a killswitch that lets you turn off a risky feature without a deploy.
Retries are not a cure, by the way. Retry blindly and all you do is pile more traffic onto a service that’s already struggling. For high-volume calls, wrap them in a circuit breaker. After a run of 5xx responses, stop calling for a while and give the other service some room to recover. Then try a single request, and only go back to normal traffic if it works.
Closed: calls go through. A run of 5xx trips it open, and callers fail fast instead of piling on. After a pause, one trial request goes through (half-open). If it succeeds the breaker closes; if it fails, it opens again.
Writes are where you have to be extra careful. Say you send “charge this user” and get a 500 back. Did the charge go through? You really can’t tell, and if you retry you might bill them twice. The standard fix is an idempotency key. It’s a UUID the client generates once for that charge and sends with every attempt. The receiving service keeps track of every key it has processed, so when the same one shows up again, it hands back the original result instead of charging again.
The client makes the key once per charge and sends it with every attempt. On the retry the service finds the key it already processed and returns the first result instead of charging again.
The other thing I’d push you to do is decide ahead of time how each part of your system behaves when a dependency disappears. Take a rate limiter that checks a Redis bucket to see whether a user has gone over quota for the current window. If Redis is unreachable, you’ve got two choices. You can fail open and let the request through, or fail closed and return a 429. For rate limiting I default to failing open, so a bug in the limiter doesn’t turn into an outage for everyone. Auth is the opposite, and it should always fail closed. Locking someone out of their own account is bad, sure. Showing them someone else’s data is far worse. A lot of cases sit somewhere in between, and for those you have to weigh what each kind of failure would cost you.
Take the dependency offline. The rate limiter fails open and lets the request through; auth fails closed and blocks it.Dependencies online. Both checks run and the request is allowed.
There’s a lot I left out on purpose: breaking a monolith into services, containers versus VMs, tracing, API design. Some of that matters less than people think (monoliths are fine for most teams). Some of it is obvious (use tracing). And API design really deserves a post of its own.
Like I said at the start, good system design mostly comes down to picking boring, proven components and putting them in the right places. Think about a plumber. If a plumber’s job ever gets exciting, something has probably gone wrong, and it’s about to get messy.
Now, the kind of design you’d show off at a conference does have its place. Every so often a custom data structure makes a feature possible that couldn’t exist any other way. But those cases are rare. The mistake I see far more often is adding infrastructure before you actually know you need it.
I saw this happen at AmaliTech. A team was building an internal app for leave and office management, so things like booking conference rooms and tracking attendance. They’d integrated with the SSO service my team runs, but for their own reasons they still had to manage refresh tokens themselves. To keep those lookups fast, they proposed putting Redis in front of the database as a cache.
Nobody had load tested anything, though. And look, caching hot lookups in Redis is a standard move, so the proposal wasn’t unreasonable. I still pushed back. The data was small, and the database they already had could handle the concurrent requests fine. We also have fewer than a thousand employees, and they’re spread across a lot of internal tools. They’re never all in the leave app at the same time. Someone books a room, closes the tab and goes back to whatever they were doing. Even if traffic hit double or triple our headcount, the database would cope. Take the extreme case, where all thousand people log in during the same minute. That’s still under 20 requests a second (1,000 ÷ 60 ≈ 17). A database barely notices that.
Even if every employee signed in during the same minute, the database would see under 20 requests a second. Each bar is one second. The proposed cache can wait for a measurement.
And Redis isn’t free. It’s one more service to run and pay for, and at a company that takes FinOps seriously, that cost needs a reason. This one didn’t have a reason yet. There were no measurements and no slow queries, only a feeling that things might get slow at some point.
If token lookups ever do slow down, the p95 numbers will show it. That’s the time to add Redis, with a measurement to justify the bill.