Zero downtime deploys for an MMORPG game server

Intro

0:00 Hello and welcome back to episode thirty-six, where we're going to deploy a new version of our worlds without throwing anyone out of them. Let's begin with a quick look at where we're going to be by the end of the episode.

0:12 Alice is playing in Ember, when Ember's next instance starts next to it, on another port. Then the old Ember is told to stop, and hands alice over to the new one before it goes, so she walks on, with her world on the screen the whole time.

0:28 On the website of the series you can find more information about the project. The code for the project is open source on GitHub, tagged at each step of each episode so that you can follow along.

Last time

0:40 Now, before we get to today's work, let's quickly go over what we did last time. Loki and Alloy joined our stack, with Alloy reading each container's output from Docker, and Loki keeping the lines for a week. Our lines reached Loki without their service at first, until loki source docker read the relabelled containers. Each line got its stream as a label, so our servers' errors stand apart from the rest. The dashboard counts the errors each minute, and lists the servers' lines, with their level. Loki's ruler emails us when the saver keeps failing, which no metric showed, and CI checks both configs.

Today

1:16 Since episode nineteen, our deploy script ships a release to the VPS, and starts each container again from it, the worlds among them. A world that stops throws its players out to the login screen, so each deploy does that to everyone playing, and we'd want to deploy often, with a fix or a new skill. So we're going to run a world's next version next to it for a moment, as a second instance of that world, on another port. When the old instance is told to stop, it leaves Consul, finds the next instance there, saves its players, and tells each client where to go, with a token to say hello there with. The client moves over without losing the world on its screen, and so do the bots. And the deploy script is going to restart each world through its next instance, so that a deploy keeps everyone playing. Blue-green deployment works like this too, where the new version starts next to the old one before the old one goes. All right, let's get started, with what a restart does to a player.

2:14 Here's alice in Ember, as it runs from the end of last episode. A deploy stops Ember, as Docker sends it the signal to stop, and alice is back at the login screen, with the connection to the server lost. Ember's log says that it stopped, and nothing about alice, who was still in it when it went.

Two instances of a world

2:32 So, in our plan, Ember runs on its port, seven seven seven seven, with alice in it. A deploy starts Ember's next version as a second instance, on seven eight seven seven, and it registers with Consul, which has to tell the two apart. The login server lists the newest, so that a player who logs in from then on goes there. Two instances of a world each run the world on their own, so two that both took players would split the world in two, and the handover has to be quick. When Docker sends the old instance the signal to stop, it leaves Consul, saves its players into Redis, where the next one loads them from, and sends each client a move, with where to go. A client says hello to a world with a token, so each move carries one, and the client says hello to the next instance with it.

3:21 In consul dot h, a world registered under its name, so a second Ember would replace the first one in Consul, and the login server would lose track of it. The registration gets when the instance started, in seconds since the epoch, so that worlds can pick each world's newest instance. And a free function, instance id, names each instance. A world runs as two instances for a moment, while one hands its players over to the other, so each instance registers under a name that's its world's and its port's. Instance id takes the whole registration, so that the server and the test both work it out from what they register. Pass and deregister take that name now: pass passes the instance's health check, with NotFound when Consul doesn't know the instance. And worlds now lists each world's newest instance, which is what the login server shows. In consul dot c c, algorithm goes, as we won't sort the worlds any more, and map and utility come in. Instance id goes at the top, outside the class, as a free function. An instance's id is its world's name, a dash, and its port, such as Ember dash seven seven seven seven. Two instances of a world on one machine have a port each, so the id tells them apart. Register uses the id for the service, and keeps the world's name in the service's metadata, with when it started, written out in digits, because Consul's metadata is strings. We could split the id at its dash instead, but a world's name could have a dash in it, so the metadata says it. Pass puts the instance's id in the path it calls on Consul, and so does deregister, so that each of them acts on one instance, and leaves the world's other instance alone. Then worlds, which listed each instance it found, and would list Ember twice while both instances run. The list we built becomes a map of each world's newest instance so far, with when it started, by the world's name. The world's name comes from the metadata now, since the service's id isn't the world's name any more. We read when each instance started too, from the metadata, into a number next to the count and the hash that the other fields go into. The check that keeps broken entries out gets two more conditions, so an instance without a start time, or with one that isn't a number, stays out, like one without its players. An instance older than the one we have for its world is skipped, and otherwise it takes the world's place in the map. Then the map's instances go into the list, which reserves room for all of them first, as we know how many there are. The map keeps the names in order, so the worlds come out sorted, which the sort did before. In main dot c c, clock dot h comes in, for absl's now. The registration starts with the time now, in seconds since the epoch, and we work out the instance's id once, for when the server stops. The heartbeat's work on Consul's thread has a copy of the registration, with the players it had at that beat, so it works the id out from that, to pass the instance's check. The line further down, which says in the log what it registered as, gets the id too, and clang-format lays it out over a few more lines. And the server deregisters the instance by its id as it stops, where it deregistered the world's name before. In Prometheus's config, the world label came from the service's id, which is an instance's now, so it comes from the world in the metadata instead. Two instances of Ember are then both Ember, and the dashboard and the alerts keep the names they had. The Consul test now checks that Consul lists each world's newest instance until that instance deregisters, and its name says so. It registers two instances of a world, older and newer. The older one becomes a variable, as the test registers it, copies it, and deregisters it by its id. It keeps port seven thousand, its metrics port, twelve players and the content hash, and it started at a hundred seconds past the epoch, a made-up time, which is fine in a test. The newer one is a copy, on another port, with no players, so that what's listed says which instance it came from, and it started later. Then both of them register with Consul, the older one first. With both instances up, the world is listed once, at the newer one, which is what the login server lists while both run: one Ember, with the newer one's port and players. Once the newer one deregisters, by its id, as a server does when it stops, we ask Consul for the worlds again, which has to come back without an error. The world is back in the list, at the older one's port, which is where a player who logs in would go then. And with both gone, the world is gone from the list, as before. And the README says how each instance registers, and which one the login server lists.

8:04 The Consul test needs Consul, so Compose starts it for the tests, and the test passes: two instances of a world are listed as one, at the newest.

Handing players over

8:19 In nanommo dot proto, a move gets a message, with a host, a port, and a token. A move is where a player goes when their world's instance hands its players over to the next one. It has the host and port players reach that one at, and a token to say hello there with, like the login server's. The world gets the move as field fifteen, and a move goes to a client alone in a world, like chat, over TCP, because it has to arrive. In server dot h, the server gets hand over, which hands the players over to the world's next instance, at the host and port players reach it at. It saves them, stops ticking and taking players, and tells each client where to go, and it returns how many it told. And the server keeps whether it's handing its players over. In server dot c c, hand over sets that first. The next instance loads each player from Redis, so they go in now, and nothing changes them here afterwards. It closes the acceptor, so a client that got this instance from the login server a moment ago is refused, and its player logs in again, to the next one. Then it saves the players that changed into Redis, like each tick does. Each player needs a token for the next instance, and the server already has the one each player said hello with, which it keeps for their hello over UDP. So the loop goes through those tokens, each with its player, and each player gets a move with the next instance's host and port, and their token, in a frame, over their session. And tick returns before doing anything once we're handing over, so that nothing moves or gets saved after the players have gone into Redis. In main dot c c, vector comes in. We give the clients ten seconds at most, which is plenty: each client leaves once it's in the next instance, within a few seconds. A timer, leaving, and a deadline go with it. Wait is a function the timer calls, so that each call can set the timer for the next one. Once the server has no players, or the deadline has passed, it stops the server, and otherwise it looks again in a tenth of a second. The signal to stop used to stop the server straight away. Now it stops the heartbeat, and waits for the Consul thread to finish what it's doing, so that nothing registers us again. Then it deregisters this instance from Consul, which the server did as it stopped, and says so if that fails. Then we ask Consul for the worlds, with this instance gone from them. A deploy starts the world's next instance before it stops this one, so with us gone from Consul, the world's newest instance is the next one. So we look for ours by its name, and next points at it, if Consul answered and our world is in its list. Without a next instance, or with nobody to hand over, the server stops as it did before. Otherwise it hands its players over, to the next instance's host and port, and logs how many it told, and to where, with player or players to fit the count. The deadline is ten seconds from now, and wait starts looking. Below io run, the join and the deregistering go, as the signal does both now, before the handover. And the README gets a section on deploying without downtime, with what a world does on the signal to stop.

A client that moves

11:30 In connection dot h, optional comes in, for the move. A connection has a getter for its move, an optional one: where the world's instance told us to go, once it hands its players over. The connection keeps it, empty until a move comes. In connection dot c c, read sorts out what comes over TCP. A world with a move in it comes first, because its chat is empty, and it would otherwise be received as a world. It goes into the connection's move instead, since it has nothing of the world in it, which the comment above now says about chat and moves both. In main dot c c, the client keeps a second connection, next, the world's next instance, while we go over to it. In the loop, after polling the connection we play on, the client checks for a move, in an if that declares it. As soon as there is one, and we aren't moving yet, it connects to the next instance, with the move's host, port and token, over the transport we play with. If that works, the connection it gets becomes next, and the client logs where it's moving to, while the move is in hand. If it can't connect, it says why, and drops the connection it plays on. The instance we're on has stopped ticking, so there's nothing left to play here. So we go back to the login screen, with the error we show for a connection we lose. We go over once the next instance has sent us our world, so that the world stays on the screen. Each frame, the client polls the next connection too, and once its world says who we are, the next connection becomes ours. Standard exchange takes it and leaves next empty, which a move would too, but this says so, where the loop reads next again on the next frame. The one before is destroyed with it, which closes it, and lets the instance before see us go. And what we sent last is forgotten, so that the next frame sends our input to the new instance again. If the next instance closes our connection instead, we go back to the login screen, as for any connection we lose, and the connection we play on is checked for that last, while no move is on its way. In server dot c c, one line goes in where a player joins. Setting broadcasts to zero makes the broadcast after a join a full one. Names, kinds and NPCs go out once a second, so a player who's just joined, say from another instance, gets them in their first world. Without it, alice would see the monsters as cubes, and no NPCs, for up to a second after a move. And the README says how the client moves.

13:58 Let's try it with alice, who plays in Ember, on port seven seven seven seven. Ember's next instance starts on seven eight seven seven, and registers with Consul a moment later. Then the first one gets the signal to stop, and hands alice over, as its log says, and alice is back at the login screen, with the connection lost. The next instance's log says why: it refused a client with an unknown token. Tokens are one-time, since episode six, so that a token someone copies is no good once it's used, and the token alice said hello with was used, by the instance before.

A token for each move

14:34 So in hand over, the loop goes through the sessions instead, by player. The server issues a token for each player in Redis, like the login server does, with its thirty seconds to live, which is plenty for a client that connects straight away. Tokens are one-time, and the player's own is used, so each move gets a token of its own. If Redis can't issue one, that player gets no move, and an error goes in the log, so they lose their connection when this instance stops, while the others still go. The move then carries the new token, and goes over the player's session. In server test dot c c, a new test hands alice over to a next instance, and pins down what the game just got wrong: the move's token has to work there. The test's second server, next, is created like the one the fixture made, with the fixture's world, content, tokens and players, on port zero, so that the system picks a free one. It's the world's next instance, in this process, on a port of its own. Alice joins the first server, with a token, and the test runs until she sees herself in its world. Hand over then tells one player, and alice gets a move, which the test keeps, to the next instance's port, where next listens. A second player, moved, connects to the move's port, and with the move's token, alice plays on in the next instance, and the one before has nobody left once she leaves it. The moved player sees alice in the next instance's world, and once alice's first connection closes, the first server's players go down to none. With the hello's token back in hand over, this test fails, as the next instance closes the moved player's connection. And in the README's section on deploying without downtime, a few lines say why each move gets a fresh token, in Redis like the login server's.

16:17 The server tests need MySQL and Redis, so Compose starts them, and the three pass, the handover among them.

16:34 With a token for each move, let's do that again. Alice walks across the Meadows while Ember's next instance starts on seven eight seven seven, and registers with Consul. The first instance gets the signal to stop, hands alice over and stops, and the next instance's log says that she joined. On her screen, the world stays, and alice walks on once the next instance's first world comes in, a moment later.

A deploy through the next instances

17:02 Now for the deploy, which runs each world's next instance in Compose. In compose dot yaml, the world server gets a health check first. A world hands its players over to its newest instance in Consul, so a deploy has to know when the next one is there, and the check asks Consul. Curl asks Consul's agent about the instance Ember dash seven seven seven seven, and fails unless Consul has it, passing. Dash f makes curl fail on an error status, such as the four oh four Consul answers for an instance it doesn't know, and dash s and dash o keep it quiet. Docker runs it at intervals of two seconds, and calls the world unhealthy after thirty failures in a row, like our other checks. The world also gets a stop grace period of fifteen seconds, the time Compose gives it between the signal to stop and killing it. The default is ten, which is all the handover waits for, so this leaves it room. Frost extends the server, so it gets the check too, with its own instance. Each world gets its next instance as another service. Server next extends the server, so it's built and set up like the server, on port seven eight seven seven. The override tag replaces the server's ports, instead of adding to them, as seven seven seven seven is taken, and it publishes its own port for TCP and UDP. Its health check asks for its own instance, Ember dash seven eight seven seven. On Compose's network, it answers to the world's name too, server, so a container that reaches Ember as server reaches either instance, which the bots are going to need. It's in a profile called next, so that docker compose up leaves it out. Frost gets its next instance too, frost next, which extends frost, on port seven eight eight seven. It has its own check, for Frost dash seven eight eight seven, frost as its other name on the network, and the next profile, like server next. In the Dockerfile, curl joins the toolchain's packages, as the sanitiser job in CI runs our servers from an image built on it, and the runtime's, which the images of our servers start from, the profiling one among them. Handover dot sh is new, and starts a world again from its newest image, such as server or frost, while its players play on. Its next instance takes them while it starts again, and hands them back. Set dash e u stops the script at the first command that fails, so a next instance that doesn't start leaves the world as it is. The script takes the world's service, and the next instance is the service with next after it. It starts the next instance first, without its dependencies, as the stack is up, and waits until it's healthy, which is once Consul lists it. Then it stops the world, and stopping a world hands its players over to its newest instance in Consul, which is the next one now. It starts the world again, from its newest image, and waits until Consul lists it, and stops the next instance, which hands the players back, to the world that's now the newest. And it removes the next instance's container, so the world ends up where it started, on its own port, in its own service. The next instance runs for a few seconds of each deploy, while it holds the world's players. In deploy dot sh, the build takes the next profile, so that the next instances' images build with the rest. Now the worlds start again through their next instances, one at a time, so that their players play on. Compose lists its services, apart from those in a profile, such as the next instances, and grep leaves out the two worlds. So the containers start again from the new release as before, apart from the worlds, which start again through handover dot sh, Ember first, and then Frost. The script stops at the first command that fails, so if Ember's handover goes wrong, the deploy stops there, with Frost as it was. In compose dot prod dot yaml, server next gets the MySQL password and the public host, like the server, and frost next gets those two as well. Neither gets a restart policy, unlike the worlds, so that a reboot starts the worlds, and leaves the next instances stopped. Alloy's rules give the next instances' lines a level, like the worlds', as their regular expressions name them too, for stderr and for stdout. And on the dashboard, the errors a minute and the servers' lines take them in too, so that a handover's lines show up next to the world's. And the README says how to run a handover, why it waits for each instance to be healthy, and that on a server, the next instances' ports are open to the internet too.

Bots that move

21:36 Our bots get the moves too, or they'd play on at an instance that has stopped ticking, until it stops. In bot dot h, a bot hands out its move, like the client's connection: where the bot's world told it to go, once it hands its players over. It gets leave as well, which closes the connection, the way a player who goes elsewhere does. And the bot keeps the move, empty until one comes. In bot dot c c, receive skips chat, as chat has no land in it, which would read as the Meadows, and neither does a move. So a world with a move in it goes into the bot's move, and receive returns before it reads any land or players from it. Leave stops the bot, and closes its TCP connection, which tells the instance before that the bot has gone. In bot metrics dot h, gauge dot h comes in, and the bots' metrics get set connected, for how many bots are connected, with the gauge it sets, next to the age histogram. In bot metrics dot c c, the gauge is built after the histogram, called nanommo bots connected, with help that says it's how many bots are connected to a world, and the transport as its label, like the world's age. Set connected sets it to the number it's given. In main dot c c, c std def comes in for size t, optional for the move, system error for the resolver's error, and our protocol's header, for the move's message. The swarm takes the host the bots play on, as a string, next to the server it resolved, and keeps it. In follow, a bot whose world hands its players over goes to the world's next instance, as a client does, with the move's token. Bots play against a single machine, so the bot keeps its host, and takes the move's port. It goes through the bots by their index, as it replaces the ones that move. It takes each bot's move, and skips each bot without one. For one with a move, the resolver looks up the bot's host, with the move's port, and puts what goes wrong in an error code, as an exception would stop the swarm. In Compose, that host is server, which both instances answer to while both run, and asio's connect tries each of a name's addresses until one answers on the port. If the host doesn't resolve, the bot says so, and stays where it is. It copies the move's token, as the bot that holds the move is about to go, and works out the bot's number, as the next bot gets that number too. The bot that moves leaves, but the bot before stays until we stop, because its handlers may be waiting on its sockets. So the swarm keeps it in left, where those handlers still find it. A new bot takes its place, with that number, and starts with the next instance's address and the move's token. Then the swarm logs how many bots moved, which is the line we're going to look for. Report sets the gauge each second, and follows the moves after the count, since a bot that moves takes a moment to connect again. The swarm keeps the host as a member, next to the bots that left. And main passes the host it was given to the swarm. On the dashboard, players online adds a world's players up across its instances, by world and land, so that Ember's players add up across both instances, instead of two lines that cross. And the README says how the bots follow a move, and what the gauge shows.

24:53 With Ember in Compose, and forty bots playing on it, let's run handover dot sh for Ember. It starts server next and waits until Consul lists it, stops the server, starts it again from its newest image and waits for that one too, and stops server next. The bots' log says that all forty moved, twice, to the next instance and back, and the last count the swarm logged, as it logs one when the count changes, is all forty connected.

Wrap-up

25:19 So, today each instance of a world registers with Consul under its own name, and the login server lists the newest. On the signal to stop, a world leaves Consul, saves its players, and sends each client a move to the next instance, with a fresh token, since reusing the hello's token sent alice to the login screen. The client moves over without losing the world on its screen. Handover dot sh restarts a world through its next instance, waiting for each instance to be in Consul, and the deploy script restarts the worlds that way, and the bots follow a move like a client does.

Playing it

26:46 Let me know in the comments what you'd like to see next. And if you want to follow along with the series, subscribe to the channel. I hope you find this useful, and thanks for watching.