Seven Minutes

From the Kitchen

Two heavy jobs came due in the same stretch of September. Move the front door. Rebuild the foundation. Both on a building full of people, both while the kitchen stayed open.

Here is the part I did not expect. The one I feared went perfectly. The one I thought was routine put a "back in ten minutes" sign on the door and left me standing there watching the clock.

Seven minutes, in the end. Not a disaster. But downtime is downtime, and I caused it on the job I respected least. This issue is about those seven minutes, the change that cost nothing right after them, and the difference between the two.

The Front Door Went First

Every score a game sends to CheddaBoards passes through a small server of mine. It signs the request, proves it came from a real, checked player, and only then will the network accept the write. Reads do not go near it, since they come straight off the chain, but writes do. And that server had lived on the same host since the very beginning.

The busiest games had started to outgrow it. Their write traffic was pushing the host's limits in a way I could not cache around, because you cannot cache a write. So the server needed a new home: a near-identical copy, stood up on a platform built for that kind of load, and tested against a staging setup before a single real request touched it.

The move was the easy part. The handover was the whole game. That server signs as one specific trusted identity, and the canister only accepts writes from an identity it already knows. Rotate the signer, and you have to teach the canister to trust the new one before you send it any real traffic, or every submission in the world bounces at once.

I got that order right. Trust first, then traffic. What I underestimated was the seam in the middle: the moment the world's requests stop arriving at the old address and start arriving at the new one. For about seven minutes across that switch, writes did not land. A game trying to submit a score in that window got nothing. Reads kept working the whole time, because reads never needed the server, but anyone finishing a run right then watched their score fail to save.

Seven minutes. I sat and watched every one of them, which is its own kind of penance.

The honest lesson is not a technical one. I had rehearsed the foundation change for days and treated this one as a formality, because moving a server felt routine next to migrating a whole persistence model. Routine is exactly the word that gets you. The seam I did not rehearse is the seam that cost me.

The job you don't fear is the one that bills you.

Then the Foundation

Which is why, when it came time to move the foundation, I treated nothing as a formality.

Quickly, for anyone who missed issue four. The CheddaBoards backend keeps its data through upgrades because of how it is built: the network remembers by default. But there are two ways it can remember, an old model and a newer one, and switching between them runs in one direction only. Back in August my compiler tried to make that switch without asking; the network refused, and I pinned the old model to get unblocked. That pin was a promise to come back. The newer model is the safer foundation, and every week I stayed on the old one was a week spent building on ground I had already decided to replace.

In September I kept the promise. And it is not a small switch: it moves every score, every account, every achievement across in a single step. It cannot half-happen, and it cannot be undone by editing a file and deploying again. Once you cross, the only road home is a snapshot you took before you left.

So I rehearsed it on a canister I am allowed to destroy. Same old model, same shape of data, same tools. I did the whole migration there, and then I practised the disaster: took the snapshot, made the cut, and deliberately rolled back, to prove the way home existed before I ever needed it.

That rehearsal is where I found the trap. A snapshot does not just save your data. It saves the whole canister, code and all, frozen at that moment. So when you load it to roll back, you restore the old code too, including the old persistence model. The rollback undoes the migration so completely that your build config, still set to the new model, is now aimed at a canister that is old again. Deploy once without thinking and you walk straight back across the one-way bridge you just retreated over. I only know that because I hit it on the throwaway, where hitting it was free.

Then the real one. Snapshot the live canister. Make the cut. Bring it back up.

Nothing happened.

No downtime. Not seven minutes, not seven seconds. I compared the system's state before and after, value by value: every count, every total, identical. The leaderboards read the same the moment after as the moment before. The change that scared me most produced no evidence, from the outside, that anything had changed at all.

The difference between this and the front door was not difficulty. The foundation was the harder, more dangerous job by a distance. The difference was that the seven minutes had already taught me the lesson, and this time I let them.

Three lessons, for anyone standing where I was:

One. Rehearse the things you cannot undo, on something you are allowed to destroy. A one-way change is only safe to get wrong in a place where getting it wrong is free.

Two. A snapshot restores your code, not only your data. So your rollback also rolls back the setting that caused the problem. Undo the change in your build config too, or your very next deploy quietly re-does the thing you just escaped.

Three, a returning guest from issue four: do not trust the deploy message. I checked the result against the live system, value by value, not against the tooling's word that it had worked. The success message is the label on the container. The running system is what is actually inside it.

The Month in Numbers

Everything else September shipped, in brief.

CheddaBoards went into review at a third marketplace: the Unity Asset Store. The Godot side has been live for a while (an addon and a full template), and in September the Unity SDK joined the queue, packaged around a tiny cheese-clicking demo that shows the whole integration in one small scene. When it clears, the same backend will be one click away in the two biggest engines their developers shop in.

The documentation grew into its own home at docs.cheddaboards.com and got a full audit: every "you need a session" claim checked against the live system, a production checklist added, a homepage that finally explains what this is. New guide pages went up for what people were arriving to search for, and the marketing copy stopped promising anything the product does not yet do.

A quieter fix I am fond of: a game submitting to several boards at once had been counting one play as several. Corrected now: one run is one play again, with a short window that folds the duplicates together without punishing a genuine fast replay. Nobody would have thanked me for it. That is usually the sign it needed doing.

And the platform kept moving with nothing pushing it: past 40,000 submissions, from more than 2,300 players, across more than 80 games. Still just listings, word of mouth, and games quietly turning up.

If you want to know when the next one lands, you can subscribe to the mailing list.

One Thing From the Kitchen

Ask any chef which dish burns. It is never the hard one. It is the one they have made a thousand times, the one so routine they stopped standing over it.

The front door was my thousandth omelette. I had moved servers before, I knew how it went, and I looked away. Seven minutes later I was reminded that the network does not care how many times you have done a thing. Only whether you were watching this time.

The foundation got my full attention because it frightened me. So it went clean. The lesson I am keeping is to give the routine job the same fear I give the frightening one.

Respect the easy plate.