Redis write-behind for a C++ game server

Intro

0:00 Hello and welcome back to episode thirteen, where we're going to make sure that a crash doesn't cost our players their progress, without slowing our server down. Let's begin with a quick look at where we're going to be by the end of the episode.

0:13 I walk off towards a corner of the board and stop. Then the game server crashes, so it can't save anything on its way out. It starts again, I log back in, and I'm where I was, as of the server's last tick.

0:29 On the website of the series you can find more information about the project. The code for the project is open source on GitHub, tagged at each step of each episode so that you can follow along.

Last time

0:41 Now, before we get to today's work, let's quickly go over what we did last time, when our game got a website. It's a Laravel app, which we made with Composer in Docker, so we didn't need PHP on our machine. And it runs on FrankenPHP, as one more service in our compose stack. Its front page ranks the players by experience, from the table the game server saves them in. Pico CSS styles it, so we wrote no CSS of our own. You can create an account on it, log in and log out. And it hashes passwords with the login server's settings, so an account made on the website logs into the game, and the other way round. Those rankings read the players table, and the game server writes a player into it when they leave, which is where today's problem starts.

Today

1:29 Today, we start with a problem our game has right now. When the game server crashes, the players lose what they did since they logged in. The obvious fix is to save each change to MySQL as it happens, so we're going to try that first, and measure it with our bots. Back in episode one, I said Redis would hold the hot state, such as positions, once saving each move directly to the database put too much load on it. So, like I said then, you get to see the problem first, and then how Redis solves it. Then we put Redis between the game server and MySQL. The game server saves into Redis, all the players in a single round trip each tick. And a small program of our own, the saver, takes them on to MySQL, a batch at a time. Then we measure again, and crash the server again, to see whether the players come back where they were. All right, let's get started with what a crash does to our game right now.

The problem

2:22 The game is as we left it last time, with a fresh database. I make an account, alice, and I'm in the middle of the board, where new players start. I walk away from the middle, towards a corner of the board, and stop. And now the game server crashes. Kill dash nine sends SIGKILL, which a process can't catch or handle, so the server stops right there. A bug that segfaults looks like this from the outside. And so does the kernel killing the process when the machine runs out of memory, or the machine going down. A new game server starts, I log in again, and I'm back in the middle of the board, where I started.

3:04 Let's go over when the game server saves, so far. When a player joins, the game server loads them from MySQL, from the players table. Then, while they play, the player lives in the server's memory, and each tick changes them there: where they are, and the experience from each monster they kill. When they leave, the server saves them, and when the server stops, with Control C or a SIGTERM from Docker, it saves everyone. A crash skips both, so everything since a player logged in is gone. Saving on leave shows on the website too, as someone who plays for hours without logging out doesn't move up the rankings until they leave.

A baseline

3:41 Before we change anything, let's measure where we are, so that we have something to compare with. Docker compose up starts the stack from the end of last episode: MySQL and Redis, the two servers, Prometheus and Grafana. We make a folder for the results, and run load dot s h, our benchmark from episode eight. And it adds two hundred and fifty bots a step for nine steps, into before dot json. Each step takes close to a minute, so this is sped up. The server keeps its twenty ticks a second up to seven hundred and fifty players, and with a thousand, it manages fifteen, and with two thousand, four. At the end of episode eight, it kept up with twenty-five hundred, and since then, it has started sending the world over UDP, in parts, and running monsters and fighting. Why that number moved is a job for a profiler, like in episode eight, and today, this is the line we hold our saving up against.

Saving each tick

4:36 The obvious fix is to save each player's progress as we go, while they play. After each tick's step, we look for the players whose position or experience changed since we last saved them, so a player standing still costs us nothing. Each of those gets the INSERT we have already, with ON DUPLICATE KEY UPDATE, so it adds a player's row the first time, and updates it after that. Then the broadcast goes out, as before, once the saving is done. Once that's in, a crash costs no more than the tick it happened in. And there's nothing left to save when a player leaves, or when the server stops, so those saves go. We're also going to time the saving, with a histogram of its own, so the dashboard can show us what it costs.

5:21 We start in players dot h, with what we're going to save. Until now, Save took a whole Player, the struct the world plays with, with its health, its target, its input and so on. Three of those fields go to the database, so they get a struct of their own, Progress. Over the struct goes a comment: what we keep of a player from one session to the next. And it has x and z, where the player is, and the experience. Then operator equals equals, set to default, like the pixel's in episode two, so two Progresses are equal when all three of their fields are. We're going to compare what a player has now with what we saved last, to tell whether they changed. Comparing floats with equals equals is usually a mistake, but here it's what we want, as saved is a copy, so any movement at all is a change. And Save takes a Progress now, instead of a whole Player. In players dot c c, Save takes the progress too, and the values for the statement come from it, while the INSERT stays as it was. In players test, the two saves pass just the progress, without the name, as a Progress doesn't have one. In server dot h, save all goes, and in its place, a private function, save changed, which the tick is going to call. And saved underscore, a map from each player's id to the progress we saved last for them. In server dot c c, in the unnamed namespace, progress of takes the three fields out of a player, as we need them in two places. And like in the header, save all goes from this file too. In handle hello, once a player has joined, what we loaded for them goes into saved, as that's what the database has. Otherwise, each player would be saved again on their first tick, before they've done anything. In leave, the save goes, and their entry in saved goes with them. In the tick, save changed goes right after the step, which is what changes the players, and before the broadcast. As it's inside the tick, the time it takes shows up in the tick's histogram too, which is what we want to see. Then save changed, which starts with zone scoped, so that Tracy shows it in the tick's timeline, if we profile it. Start is the time now, so that we can time the saving for the metric. Then, for each player, progress is what they have now, and saved is a reference to what we saved last, so that we can update it in place. If the two are equal, the player hasn't changed, and continue skips them. Otherwise, we save their progress, and keep the status that comes back. If that fails, we print the error and carry on, and as their saved stays as it was, the next tick tries them again. If it worked, saved becomes the new progress, and at the end, we observe how long all of it took. Last, in main dot c c, the call to save all goes, and the line we print on the way out says stopped. Let's look at the loop again: with a few hundred bots walking around, most players change in each tick. So this is going to run an INSERT for nearly all of them, twenty times a second.

8:19 Then the metric, which tells us how long the saving takes. In metrics dot h, observe save goes next to observe tick, and save seconds, a second histogram, next to the tick's. In metrics dot c c, an unnamed namespace, with time buckets, a function that returns the bucket boundaries from episode four. They go from a tenth of a millisecond to a second, closest together around the fifty-millisecond budget, and both histograms are going to use them. It's a function and not a global constant, as the style guide doesn't want globals that need a destructor, and a list of numbers in a vector does. The tick's histogram takes its buckets from time buckets now, like the saving one is going to. Then save seconds, named nanommo save duration seconds, with the help text: time spent saving players, in each tick. It's a histogram like the tick's, as an average would hide the slow saves, and the slow ones are what make a tick late. Observe save turns the duration into seconds, like observe tick does. In the dashboard, a new panel, saving players, with the p50 and the p99 of that histogram, and the fifty-millisecond budget as a dashed line. And load dot s h asks Prometheus for the p99 of saving too. Saves is the sum of its bucket rates by l e, like the tick's quantile above it. The p99 comes in through arg json, like the others, so it's a number in the result, as save p99 m s, which our charts are going to read. And the line printed for each step shows it, after the tick's p99.

9:54 Let's measure what saving after each tick does to our server. Docker compose up starts the stack from this step, and this time, the benchmark adds fifty bots at a time, for eight steps, into mysql dot json. I don't expect this to go far, so it's fifty a step, and not two hundred and fifty like before. With fifty players, the server manages eighteen point eight ticks a second, where it managed twenty with seven hundred and fifty before. The tick's p99 is about one hundred and twenty milliseconds, and nearly all of it is saving. With a hundred players, it's down to eleven ticks a second, and with four hundred, three.

10:31 The charts show the numbers from mysql dot json, with fifty more players at each step. On the left, the tick's p99, and the p99 of saving, as the players arrive. The two lines lie almost on top of each other, and they're past the fifty-millisecond budget from the first step. They grow with the players, as each player who moved is one more INSERT. On the right, the ticks per second, against the twenty we aim for. Before today, this server kept up with seven hundred and fifty players, and saving each change to MySQL, it can't keep up with fifty.

11:06 The dashboard shows it too, here with a hundred bots. Ticks per second is red, as the server is well under twenty. And the new panel, saving players, sits above the dashed line, the budget. The p50 is well over the budget too, so it's the typical save that's this slow, and not a few unlucky ones.

11:25 So let's look at why saving to MySQL is this slow. Each INSERT is two round trips to MySQL, as our database run prepares the statement first, and then executes it, and the game server waits for MySQL's answer each time. And as our code doesn't start transactions, MySQL runs each statement as a transaction of its own, which it commits right away. When InnoDB, MySQL's storage engine, commits, it writes its log to disk and waits until the disk has confirmed the write. That wait is called an fsync, and it's what makes a commit durable, so a committed row survives a power cut. I tried turning that off, with a MySQL setting, innodb flush log at trx commit, and saving got a little quicker, while most of its time stayed. So most of the time goes on the round trips. With fifty players, the median tick took about thirty-eight milliseconds on my machine, which is about zero point eight milliseconds for each player. And our game server has one thread, so the tick waits for all of them, one after the other. There are other ways to go about it, so let's look at two. We could put all the changed players into a single INSERT each tick, which would be two round trips and one commit a tick, and a lot quicker. But the tick would still wait for MySQL, twenty times a second, and MySQL is shared with the login server and the website. So a slow query there, or a table locked for a migration, would make our tick late. We could hand the saving to a thread of its own, so the tick doesn't wait. But then what's waiting in that thread's queue is in the game server's memory, and a crash loses it, which is the problem we started with. We want a place outside the game server that's quick enough to take each player's changes twenty times a second. It has to keep them on disk too, and let MySQL catch up from it at its own pace. Redis can do all three, so that's what we're going to use.

Redis

13:17 We've had Redis in our stack since episode six, for the login tokens. It keeps its data in memory, so a write takes microseconds, and it can write that data to disk too. For today, two parts of its docs matter, and the first is pipelining. Without it, a client sends a command and waits for the reply before it sends the next, so each command costs a round trip time, R T T. Even on one machine, that's tens of microseconds, and with a few hundred commands a tick, it adds up. With pipelining, the client sends all its commands, one after the other, and then reads all the replies, so a few hundred commands cost a single round trip. Redis also reads them with fewer system calls that way, so it gets through more commands a second.

14:03 The second is persistence, and by default, our Redis writes a snapshot of its data now and then. In our compose file, its data isn't on a named volume either, so a docker compose down loses it. Appendonly yes turns on the append-only file, or A O F, where Redis appends each command that changes the data, and when it starts, it plays the file back. It rewrites the file in the background from time to time, so that it doesn't grow forever. And appendfsync says how often Redis makes the disk write it out. Always fsyncs after each write, which is safe and very slow, like our commits. Everysec, the default, fsyncs once a second, so a crash of Redis, or of the machine, loses a second at most.

14:49 So, let's go through the plan for putting Redis between the game server and MySQL. After each tick, the game server saves the players who changed, into Redis, as a hash for each player. A hash is Redis's map from field names to values, so player colon seven has an x, a z and an experience. All the players who changed go in one pipeline, so it's one round trip a tick, however many there are. Their ids go into a set, the players that MySQL doesn't have yet. Then a program of our own, the saver, takes the ids out of that set on a five-second timer, and reads those players from Redis. And it writes them to MySQL, five hundred to an INSERT. That's a commit for each five hundred players, once in five seconds, instead of a commit for each player, twenty times a second. When a player joins, the game server reads them from Redis first, as it's newer than MySQL, and from MySQL if Redis doesn't have them. Writing to a fast store first, and to the database behind it later, is called write-behind. MySQL is a few seconds behind the game, and the game server doesn't wait for it. We're going to build it in two steps: Redis first, and then the saver.

15:59 We start with pipelining, in our Redis class, from episode six. In redis dot h, under Command, a new function, Pipeline, which takes a list of commands, each a list of words. It returns a status or a list of lists of strings, as some of the commands we're going to send reply with one value, and some with a list of them, and a list covers both. Then we go back up and write what it does: sends all the commands before reading any reply, so they share a single round trip. Each reply comes back as a list of values, empty for nil, which is how Redis says there's nothing there. In redis dot c c, the unnamed namespace with reconnect if lost, from episode six, gets a few helpers that Command and Pipeline are going to share. Reply is the unique pointer Command held its reply in, from episode six, with free reply object as its deleter. So a reply gets freed whichever way we leave the function. Command had that type written out inline, decltype and all, and a using alias gives it a short name, so the two functions can share it. Append adds a command to the context's output buffer, without sending it. Its body moves over from Command: first the pointers to the words, then their lengths. And the comment moves with them: each word goes with its length, so it can hold any bytes, spaces and all. Redis append command argv is hiredis's call for that, and it copies the words into the buffer. So our vectors can go away as soon as it returns, and hiredis sends the buffer when we ask for the first reply. Next reply sends what's buffered, and waits for the next reply to come back, so that's where our thread waits on the network, once for each round trip. Inside it, a void pointer, to hold whatever comes back. Redis get reply fills it in through a pointer to it, which is how C hands back a value of any type. If that fails, the connection is closed, and we say so, with hiredis's description of the error. Otherwise, the reply goes into a Reply, our unique pointer, which frees it for us. Text turns a single reply into a string: the number for an integer reply, and the bytes for the others, or an empty string for a nil. Hiredis keeps a reply's bytes and their length apart, as a reply can hold any bytes, zeros among them, so the string is built from both. Command becomes append, then next reply, which is what hiredis's redis command argv did for us. The reply is a reference to what next reply returned, so the switch on the reply's type stays as it was. Then Pipeline, the function we declared in redis dot h. Like Command, it starts with reconnect if lost, so a pipeline after Redis restarted re-connects first. If Redis is still away, we hand that error back before anything goes out. Then it appends them all, so they go out together, with the first next reply. A pipeline isn't a transaction, so another client's commands can run in between ours. That's fine for what we're going to send, as each command is about one player. If we needed all of them to happen together, or none, Redis has MULTI and EXEC for that, and they can go in a pipeline too. Then it reads a reply for each command, in the order the commands went out. A reply can be an error, say for a command with the wrong arguments, or for one that's used on the wrong type of key. We keep the first error, with Abseil's status update, which keeps the status it has unless that's OK. Then continue goes on to the next reply, as an error has no values to keep. And over it, we explain why: we read on after an error, or the replies left over would come back as the replies to the next commands. Each reply that isn't an error gets its list, which emplace back adds to the replies, and hands us a reference to. For an array, each element goes into the list, through text. A nil leaves the list empty, and anything else is a single value. At the end, the first error if there was one, or the replies. So that loop reads a reply for each command, even after an error. It's the part of a pipeline that's easy to get wrong, and the bug would show up later, as a wrong reply to some other command, far from this code.

20:19 Now the Players class, which is going to save into Redis instead of MySQL. In players dot h, map, from the standard library, and our redis dot h. Players takes Redis as well as the database, and keeps a reference to each. Load's comment says where a player comes from: from MySQL, with what Redis has on top, as that's newer. And Save takes all the players who changed in a tick, a map from their id to their progress, so it can send them together. They go into Redis, all of them in a single round trip, as its comment says. In players dot c c, map, and an unnamed namespace with key, which gives the name of a player's hash in Redis. And the name is player colon, then the account id, with a colon like the token keys from episode six. In Load, the query on MySQL stays as it was, and saved takes x, z and the experience from its row, after the name. Then we ask Redis for the player's hash, with HMGET, which reads the fields we name, x, z and experience, in that order. A hash Redis doesn't have gives three nils, which our pipeline turns into empty strings. So if the first value isn't empty, Redis has the player, and saved takes Redis's values instead. The parsing after it reads saved, whichever side it came from. Then Save, which doesn't touch MySQL any more, so the tick no longer waits for it. It builds a command for each player who changed: HSET, the player's key, and the three fields with their values. HSET creates the hash if it isn't there, and sets the fields we give it. Abseil's str cat turns the numbers into text, with six significant digits for a float, so two and a half is stored as two point five. Our positions stay within five and a half metres of the middle, so six digits keep them to a hundredth of a millimetre. Then all of them go through the pipeline, and we return its status. In server dot c c, we include map, and save changed changes shape. Instead of saving each player as we find them, it collects them into changed. Not equals comes for free from the equals we defaulted, so a player whose progress differs from what we saved goes in. Then a single save, for all of them, and if it worked, each of them goes into saved. If it didn't, we print the error, and the next tick tries them again, as saved still has their old progress. We observe the metric as before, so the saving panel on the dashboard keeps working. In main dot c c, players gets Redis too, next to the database it had already. In players test, the fixture gets Redis too, from test services, and skips the test unless both are there. Tear down deletes the test's hash from Redis, so that the tests don't leave players behind. And the saves pass a map with a single player in it, which is what the double braces are. Last, in compose dot yaml, Redis gets a command, redis server dash dash appendonly yes, which turns on the A O F. And a named volume, redis, mounted at slash data, which is where the Redis image keeps its files. The volume joins mysql in the list of volumes at the bottom.

23:34 Let's look inside Redis, while the game server saves into it. Docker compose up starts the game server from this step, with MySQL and Redis, and twenty bots join it. Docker compose exec runs redis C L I in Redis's container, like in episode six. HGETALL player colon one lists bot one's hash, each field followed by its value: x, z and the experience. A moment later, x and z have changed, as the bot walks, and the game server saves it each tick. And in slash data, the A O F has a folder of its own, with a base file, the file Redis appends to, and a manifest that lists the two.

The saver

24:14 Now the saver, and for it, the game server is going to add each player it saves to a set, players colon unsaved. A set holds each id once, however many times it's added, so a player who walked for five seconds is in it once. Once in five seconds, the saver flushes the set, which means saving its players to MySQL. First, SCARD counts the set, and we save that many and stop, as our bots change all the time, and without a count, the saver could keep going forever. SPOP takes up to five hundred ids out of the set, at random, and Redis does it as a single step, so no id is lost in between. HMGET reads each of those players, all in one pipeline, and a single INSERT writes them all, with five hundred rows of values, and ON DUPLICATE KEY UPDATE. If MySQL fails, the ids go back into the set, so the next flush tries them again. Then the next five hundred, until we've done the count. Five hundred to an INSERT keeps each statement, and each transaction, short, so MySQL can serve the login server and the website in between.

25:21 In players dot h, k batch, five hundred, how many players go in an INSERT. Save's comment gets an addition, and marked unsaved, as it's going to add the players to the unsaved set too. And a new function, flush, which is the saver's whole job. What comes back is a status or an int, the count, which the saver is going to print in its logs. Its comment says flush takes the unsaved players from Redis to MySQL, a batch at a time, and returns how many it saved. In players dot c c, we include algorithm, for min. In the unnamed namespace, k unsaved, the name of the set, players colon unsaved. Key now takes an Abseil alpha num, the type that str cat takes for each of its pieces, which is a number or text. Save has ids as numbers, and the saver is going to have them as text, from Redis, and key takes both. In Save, if nothing changed, we return right away, as Redis refuses an SADD with no members. Unsaved starts the SADD command, with the set's name, and each player's id goes on the end of it. After the loop, it goes in the pipeline after the HSETs, so it's still one round trip. Then flush, which the saver calls once in five seconds. SCARD counts the set once, before the first batch goes out. The comment over it says it's as many as are unsaved now, and whoever changes meanwhile waits for next time. Left takes that count, and if Redis didn't answer, or the count isn't a number, we return an error that says so. The count says how many to save, and not which ids, as the set keeps changing while we work. Then, while there are players left, SPOP takes the smaller of what's left and k batch out of the set. We write down why that's safe: SPOP takes them out of the set, so one who changes after this is marked unsaved again, and goes in a later batch. SPOP with a count takes several members at once, and hands them back as a list, which is why our pipeline returns a list for each reply. If the set is empty already, we stop, and otherwise, left goes down by how many we got. So far we have nothing more than their ids, and Redis has their progress, in their hashes. So we read those next, with an HMGET for each id, like the one in Load, with each player's values in the order of the columns. The pipeline sends them together, so five hundred reads cost one round trip. If Redis fails here, we return its error, and the ids we popped are gone from the set by then. However, the players' hashes are still in Redis, and the game server marks them again as soon as they move. Then the statement, which starts as the INSERT we had, up to values, with IGNORE after INSERT. IGNORE turns a row that MySQL refuses into a warning, and MySQL goes on with the rest of the rows. Say a player's account was deleted while they were in the set. Their row breaks the foreign key on the account, and without IGNORE, the whole INSERT would fail, the batch would go back into the set, and each flush after it would fail on that player again. So the comment over it says that IGNORE skips a player whose account is gone, with a warning, so that one row MySQL refuses doesn't hold back the batch on each flush. Rows counts the players that go in. If the first value from a player's hash is empty, the hash is gone, say because a test deleted it, so we skip that player, and the comment says why: a player whose hash is gone has nothing to save. With IGNORE, their empty values would go in as zeros, over what MySQL has saved for them. For each of the others, Abseil's str append adds four placeholders in parentheses to the end of the statement, with a comma before all but the first row, and rows goes up by one. With five hundred players, that's two thousand placeholders, well under the sixty-five thousand that MySQL allows in a statement. Values gets the id, then the three values Redis gave us, in the order of the columns. The values stay text, as they came from Redis, and MySQL turns them into numbers, like it did with ours before. When nobody in the batch is left to save, we go on to the next batch, as an INSERT needs at least one row. Then as saved, on duplicate key update, so each existing row takes the new x, z and experience. Database run sends it, with the values apart from the SQL, like all our prepared statements. If that fails, again is an SADD with the ids we took out, back into the set, so that the next flush tries them again. We send it through the pipeline, and ignore error says that we know this can fail too, and that there's nothing more we can do about it here. Then we return MySQL's error, and the players wait in the set until MySQL is back. Otherwise, saved goes up by the rows we sent, and once the loop is done, we return it. With two thousand players walking around, a flush is going to be four batches, and so four INSERTs, each with its own commit. In players test, we include vector, for the row that the new test compares. Tear down turns the id into text once now, and as well as deleting the test player's hash, it takes them out of the unsaved set, with SREM, and player zero's hash and id too, so the test leaves nothing behind in Redis. And a new test, flushes the players to MySQL. It starts by saving our player at one and a half, two, with seventy experience. Then a second player, with the id zero, and the comment over it says what that one is for: no account has the id zero, so MySQL refuses this one, and it holds back nobody. Then we flush. Flush has to work, and save at least our player, as the set is shared, and a game server playing on this Redis would have its players flushed too. Then we read the row directly from MySQL, with a SELECT, so that Redis can't answer for it. Load would read Redis first, and the test would pass without the flush doing anything. There has to be a single row, and we expect one point five, two and seventy, as text, which is how MySQL prints those numbers.

31:22 Then the saver, which is a program of its own, so it gets a main of its own, in saver main dot c c, like the login server. Its headers are ones our servers use already, starting with chrono, for the five seconds between flushes, and for timing them. Then C signal, functional, iostream, memory and utility, and Abseil's status or, and Abseil's time, for format duration. Then Asio's io context, its signal set and its steady timer. And four headers of our own: the database, players, redis, and settings, which reads the hosts from the environment. In an unnamed namespace, k interval, five seconds, the time between flushes. The interval is how far MySQL can fall behind the game, and how much work a flush does. A shorter interval would make the rankings fresher, and keep MySQL busier with smaller batches. Flush, the saver's own function, calls players flush, and times it with the steady clock, which can't jump when the time of day is corrected. If it failed, we print the error and carry on, as the next flush tries again. If it saved anyone, we print how many, and how long it took, with Abseil's format duration. When nobody changed, it prints nothing, so the logs don't fill up with lines that say zero. Main connects to MySQL first, like the login server does. Database create takes the host and the password from the environment, with defaults for running it on our machine. It connects right away, so a wrong password stops the saver as it starts, and not five seconds later, at its first flush. If it can't connect, it prints why, and returns one, so the container stops with an error, and docker compose ps shows that it exited. Otherwise, the database moves out of the status or, into a unique pointer that lives until main returns. Then Redis, which the saver needs as much as it needs MySQL, as that's where the players come from. Its host comes from NANOMMO REDIS HOST, on Redis's default port, and if it can't connect, it prints why and stops too. Otherwise, redis moves into a unique pointer of its own, like the database. These lines are in their third main now, after the game server and the login server. And if a fourth one comes along, they're going to get a function of their own. Then it makes a Players with both connections, MySQL and Redis. It's the class the game server uses, so the key names and the set's name are written in one place, for both programs. Then it prints how often it saves to MySQL, before the part that runs the flushes. An io context, like in our servers, which is going to run the timer, and wait for the signals. The timer is Asio's steady timer, the kind our game server ticks with, set to go off in five seconds. Tick is the function that runs when the timer goes off. It's a std function, and not a plain lambda, as its body refers to tick by name, to set up the next wait. It captures everything by reference, as all of it lives in main, and main outlives it. If the wait was cancelled, it returns without flushing or waiting again. Otherwise, it flushes, moves the timer on by five seconds from its last expiry, so the flushes don't drift, and waits again. SIGINT and SIGTERM stop the io context, like they do in our servers, and io run returns. And on the way out, a last flush, so that what's unsaved reaches MySQL when we stop the saver on purpose. In the server's CMakeLists, a new executable, nanommo saver, from saver main. It links the core library, which has Players, the database and Redis, and Abseil's time, for format duration. In the Dockerfile, a saver stage, like the login server's, with the binary from the test stage. In compose dot yaml, a saver service, built from that stage, with the hosts of MySQL and Redis, which are their service names. And depends on, with the condition service healthy, so it waits for both, like the servers do. And CI builds the saver's image, along with the others.

35:26 Docker compose run tests builds the tests and runs them, next to MySQL and Redis, with a gtest filter for the players tests. All three pass: a new player starts in the middle, a player loads where they were saved, from Redis, and the flush takes them to MySQL.

Measure again

35:47 Now docker compose up starts the stack from this step, with the saver next to the game server. The players come two hundred and fifty at a time again, for nine steps, up to twenty-two hundred and fifty players, with the results going into redis dot json. Saving's p99 is two point five milliseconds with two hundred and fifty players, and around ten with a thousand. And the server keeps up with seven hundred and fifty players, like it did before today. And the saver's logs, from the end of the run: each flush saved around seventeen hundred players, in about thirty milliseconds, once in five seconds.

36:22 The charts show the numbers from redis dot json, and from before today, side by side. On the left, saving's p99, through Redis, and mind the players along the bottom. They go up to two thousand two hundred and fifty this time, where saving to MySQL fell behind at fifty. Saving stays well under the budget, with three point four milliseconds for five hundred players, as it's one round trip a tick, however many players there are. It still grows with the players, to about twenty-seven milliseconds with two thousand, as each one is more to send, and more for Redis to write. On the right, the ticks per second, before today and through Redis, and the two lines lie on top of each other, so saving through Redis costs us next to nothing. The server still falls behind past seven hundred and fifty, where it did before today, so whatever limits it there, it isn't the saving.

37:15 On the dashboard, with five hundred bots, ticks per second is green, at twenty a second. And saving players is a thin line at the bottom of its panel, well under the dashed line.

37:25 And now the crash again, with Redis and the saver running behind the game server. I make an account, alice, and walk off towards a corner of the board, and stop there. Kill dash nine sends SIGKILL again, so the game server stops right there. A new game server starts, I log in, and I'm where I stopped. The last tick before the crash saved me into Redis, and the new server read me from there. Within five seconds, the saver has me in MySQL too, so the website's rankings have me as well.

Docs

38:01 Last, the README, where running the game locally now starts the saver too, next to MySQL and Redis. Redis gets a volume of its own, like MySQL, so docker compose down keeps the players, while down dash v deletes them, along with the accounts. The part on logging in says where a joining player comes from: the name from the accounts table, and the rest from Redis, or from the players table when Redis doesn't have them. Experience is saved with where you are, while health still isn't, so you join at full health. A new section, saving players, says where a player's progress is at any moment: in Redis after each tick, and in MySQL within about five seconds. Five seconds is also how far behind the website's rankings can be. It ends with three commands to look inside: a player's hash, the count of unsaved players, and the saver's log. The list of images gets the saver's, and the metrics and the benchmark now cover how long saving takes in each tick. The fifty players in the README come from my machine, so on yours, that number is going to be different.

Wrap-up

39:03 So, today we started with a crash that lost the players' progress since they logged in, as the game server saved them when they left. Saving each change to MySQL fixed that, and took the server below twenty ticks a second at around fifty players. Each save was two round trips and a commit that waits for the disk, and the tick waited for all of them. So the game server saves into Redis instead: a hash for each player, and all of them in one pipeline a tick. Redis's A O F keeps them on disk, with an fsync each second. And the saver takes the unsaved players on to MySQL once in five seconds, five hundred to an INSERT, which is write-behind. Measured again, saving takes a few milliseconds a tick with hundreds of players, and the server keeps up with as many players as it did before. After a crash, the players come back where they were, and tests pin down both halves, Redis and the flush to MySQL.

Playing it

41:02 Let me know in the comments what you'd like to see next. And if you want to follow along with the series, subscribe to the channel. I hope you find this useful, and thanks for watching.