Three Weeks

From the Kitchen

In a kitchen you count the night in covers: one cover, one diner served. A quiet Tuesday might do forty. A good Saturday, a few hundred. Nobody plans for the week where the covers triple and keep going.

Every restaurant has had the table. It books for four and orders for twelve. It keeps ordering. By nine o'clock half the tickets on the rail are theirs, and the kitchen is running to their rhythm whether it planned to or not.

You do not turn that table away. That table is the best thing that has happened to you all week. But you do find out, fast, which parts of your kitchen were only ever built for a quiet night.

In October, CheddaBoards got its busy table, and three weeks of covers like nothing before them.

Most of the Kitchen

Last issue's numbers ran to the end of September: past 40,000 submissions, nine months after launch. The exact count on the 30th was about 44,000.

Twenty-three days later it is past 143,000.

More than twice as many scores in three weeks as in all the time before.

Players went from about 2,400 to nearly 5,900. Games went from 88 to 120. No launch, no marketing, no post that went viral. Just more games, and the games that were already here getting busier.

Since the server moved house in September, its logs are actually readable. I can filter them by game, by route, by result. So in the middle of October I sat down and asked a simple question: who is sending all this?

The answer was mostly one game. A live mobile game on Google Play, built by a developer I have never met, accounted for roughly 85% of everything reaching the server in the week I looked at, the 10th to the 17th of October. Not 85% of games. 85% of requests.

It is worth being precise about that, because a request is not a score. Every time a run ends in that game, it makes a few calls: start the session, submit the score, read the boards back. About 3.3 requests per game over, on average. Most of what the server was doing, at any given moment, was serving that one game's players finishing their runs.

And the server is not the whole kitchen. Games on recent Godot and Unity SDKs read their leaderboards straight from the canister, so those reads never touch my server or its logs. That one game still reads through the server, which makes its share of the server look bigger than its share of everything. The canister's own counter says it has answered more than 430,000 read queries in its lifetime. It cannot tell me which came through my server and which came straight from a game. What I do know is that it stood at around 150,000 on the day direct reads shipped.

Which is exactly what I built this for. Somebody shipped a real game to a real store, picked CheddaBoards to run its leaderboards, and the players turned up. That is the dream version of this whole project, arriving quietly in a log file.

And then I found out what my own kitchen had been doing to it.

The Door I Built Myself

Every game's API key has a rate limit. On the free tier it was a thousand requests an hour. I set that number months ago as a safety rail, there to stop a runaway script or a bot from hammering the canister, and set high enough that no normal game would ever notice it.

A normal game would not have. A popular one did.

From the 11th of October, at evening peaks, that game was going over a thousand requests an hour. When it did, the server started saying no. And because of the order a game makes its calls in, the request most likely to be refused was the score itself, the one at the end of the run. For about a week, at the busiest hours, some real players finished real runs and their scores did not save.

I did not know, and the only reason I found out is a log line I added on the 14th for a completely different problem: a game whose forgotten test build was submitting a score on every single tap. The fix for that was to start logging every time the server turned a request away, and why. Three days later, those new lines were full of a game I had nothing against.

The fix was a matter of deciding what the limit was actually for. The thing worth protecting is the canister, and what costs the canister is writing: a score, an achievement, a name change. Reads and session bookkeeping are cheap. So the limit now counts writes only. The same game, at the same peak, now sits comfortably inside it. Nobody had to change a line of their game, and the safety rail is still there for the thing it was meant to stop.

But I want to be straight about it. For a week, my own infrastructure dropped scores from the most successful game on it, and it was not a bug in the usual sense. It was a decision I made, working exactly as designed, against a kind of success I had not designed for.

A limit a real game can hit is not a limit. It is a bug with good intentions.

What a Score Costs

The busy table showed up somewhere else too: the cycles bill.

Every canister burns cycles to run. In October the question was a quiet one, but a big one. The balance was going down faster than I had expected, and it was obvious that "faster than expected" would only get worse with every game that turned up.

So I went looking at what a single score submission actually costs, and found the usual things you find when you look properly for the first time. Work being done on every submit that only needed doing occasionally. Events being logged that nobody reads. Lookups walking the long way round because there was no index for the short one.

One canister release later, rehearsed first on the spare canister as always, the cost of a submission came down from roughly 130 million cycles to roughly 51 million. About 60% off, measured from the balance itself rather than a benchmark, on a morning's worth of real traffic. I am still watching it across busier hours. But again: nobody had to change anything in their game, and none of them would know it happened.

That is a theme I keep coming back to. The best infrastructure work is the kind where the only evidence is a number on my side of the counter.

My Own Game, Out the Door

Meanwhile, I have been on the other side of the counter.

Chedz vs the Graters is my own game: a one-button auto-runner where you jump, double jump, and squash the graters trying to shred you. It has been sitting on chedda.games as a beta for a while. In October it came out of beta, got a new menu and a proper identity, and became the featured game on the site.

Then it went further than any of my games have before. I registered a Chedda Games publisher account on Google Play, went through verification, built the Android version, and on the 25th it went live on Google Play. It is on itch.io too, and iOS is next.

The part I care about most: it is one game on every one of those stores, sharing one set of leaderboards. A player on their phone and a player in a browser are competing on the same board, and a player who links their account carries their scores with them. That is the thing CheddaBoards is supposed to make easy. It turns out that the best way to check whether it is actually easy is to do it yourself, from scratch, as a developer filling in store forms at midnight.

It also means I am now a developer on my own platform with a game on Google Play. The same position as the busy table. Which is a useful thing to be, the month you find out what your own limits do to a popular game.

The Month in Numbers

Everything else October shipped, in brief.

SDK 2.3.1 went live on all four Godot listings. It fixes a quiet one: on a shared device, if a game switched from one player to another, the second player could inherit bits of the first player's state, and a brand-new player's name change could be lost. The Unity SDK has the same release on GitHub, and its Asset Store submission is still working through the review queue. Then 2.3.2 landed at the end of the month, fixing a Setup Wizard bug that could lose the credentials it had just written if the script was open in the editor.

Security got a lot of attention. An outside review of the public backend code found a way two developer accounts could, in principle, end up sharing an owner identity. I found no evidence it was ever used, fixed it the same day, and wrote it up at the top of the public changelog, with a security policy alongside it so the next person who finds something knows where to send it. A later release tightened what the public game listings reveal about who owns a game.

Then, on the 25th, the last closed piece went public. The server this whole issue has been about is now open source, alongside the backend and the SDKs, so the whole CheddaBoards stack is out in the open. Before it went out I ran a full security audit across both. The worst find was a sign-in setting that, if a developer had ever turned it on, could have been used to log in as someone else. No game ever had. It is gone now, and that is in the changelog too.

Player stats got fixed too. Per-player play counts and average scores had always read zero for most games, because only one of the two submit routes ever recorded them. Both do now.

The docs gained a page on how player names actually work, checked against the code rather than my memory of it, plus a version written for AI coding assistants, since a growing share of developers integrate with one open next to them. And the server's logs now keep their history past the usual thirty days, so next time I ask "who is sending all this?", I can ask it about any month, not just the last one.

All told: past 143,000 submissions, from more than 5,800 players, across 120 games. Still no marketing spend. Still listings, word of mouth, and one very busy table.

If you want to know when the next one lands, you can subscribe to the mailing list.

One Thing From the Kitchen

When the busy table is in, a bad kitchen gets defensive. Slows the tickets down. Quietly runs out of things. Tries to make the table order less.

A good kitchen does the opposite. It looks at what is actually slowing it down and fixes that, mid-service if it has to, so the table never feels the strain.

The table is never the problem. The table is the reason you have a kitchen.

Never punish the busy table.