MMO server discovery with Consul and Docker

Intro

0:00 Hello and welcome back to episode twenty-one, where we're going to run more than one world, and have the client find them through Consul, without a change to its configuration. Let's begin with a quick look at where we're going to be by the end of the episode.

0:15 The client lists the worlds that are up, from Consul, and I pick Frost, where nobody has played yet, and create my account, which works in each world. Frost is a world of its own, with an island from another seed and players of its own, and a world we start shows up in the list within seconds, with nothing to configure.

0:37 On the website of the series you can find more information about the project. The code for the project is open source on GitHub, tagged at each step of each episode so that you can follow along.

Last time

0:48 Now, before we get to today's work, let's quickly go over what we did last time. Our bots started taking trips across the island, so that they spread out like players do. The server sorts the players and monsters into cells of thirty-two metres, and each player gets what's in the cells around them, which is called interest management. A profile showed that most of the rest of the time goes on the kernel sending each packet, so we cut the bytes we send. Each player sees the others within sixty-four metres, and gets their names and levels once a second. With that, a world server kept its twenty ticks a second with twelve hundred players on my machine, the most our benchmark added, and with about seven hundred on the VPS, which is where today begins.

Today

1:30 Today, we're going to run Consul, a service registry from HashiCorp, next to our servers. Each world server registers with it, and keeps telling it, each two seconds, that it's still ticking. The login server asks Consul which worlds are up, so that the client can show a list of worlds before you log in, and you pick one. Then a migration gives each world its own players, and we start a third world with a single command, which shows up in the client and on our dashboard, with nothing else to change. All right, let's get started with why we'd want more than one world.

More than one world

2:04 Even with interest management, a world server has a limit on its players, and on a small VPS, that was about seven hundred last time. Most MMOs don't try to fit all their players into one world. They run many copies of it, each with its own players, and you pick one when you start playing. Games call them worlds, servers, channels or shards, and shard is a name that goes back to Ultima Online. For us, a world is going to be a game server of its own, with a name, an island from its own seed, and its own players. The accounts stay shared, so that one login works in each world, while your character in one world has nothing to do with your character in another. Right now, the client takes a host for the login server, and connects to the game server on that host, on a port that's written into its code. So a second world would need a new client, or a setting that each player changes by hand.

Finding the worlds

2:58 So, the client needs a way to find the worlds, and the simplest is a list of worlds in the client's configuration. That means a new client for each new world, and a list that's wrong as soon as a world goes down. A table in MySQL that each world writes its address into is closer, though a world that crashes leaves its row behind, and players get sent to a server that's gone. DNS can give several addresses for one name, but it doesn't tell us whether a world is healthy, or how many players it has. Production systems use a service registry for this: each service registers with it, with a health check, and the others ask the registry for the services whose checks pass. Consul is one of them, and etcd and ZooKeeper are others, while Kubernetes does this for its own containers. We're going to use Consul, with what it calls a TTL check, for time to live. Each world tells Consul that it's alive each two seconds, and when ten seconds go by without that, the world turns critical, and drops out of the list. That covers a world that crashes, and a world whose tick loop gets stuck, as the heartbeat comes from the tick loop.

4:06 Consul is made by HashiCorp, the company behind Terraform and Vault, and it's on GitHub, written in Go. Among the features its README lists, we need service discovery: services register with it, and other services find them there, through DNS or HTTP. Health checking goes with it, so that traffic doesn't go to a service that's unhealthy. Consul does a lot more, like a service mesh, and configuration for applications, which we don't need today. We're going to run it in Docker, from its official image, and talk to it over its HTTP API.

4:39 Its documentation lists the checks Consul can run on a service, such as calling an HTTP endpoint, opening a TCP connection, or running a script. A TTL check works the other way around: Consul waits for the service to report its state, and marks it critical if no report comes in time. That suits a game server, as a port that's open says little about whether its tick loop still runs.

Registering a world

5:03 Consul's API speaks JSON, so we need a JSON library, and we're going to use nlohmann slash json, by Niels Lohmann, which a lot of C++ projects use. Its first design goal is an intuitive syntax, so that JSON feels like part of the language, the way it does in Python. For example, an object can be written as braces of keys and values, nested like the JSON it makes, which is how we're going to write a world's registration. It also supports JSON pointers, which are paths into a JSON value, with slashes between the keys, and that's how we're going to read what we need out of Consul's answers. It's a single header, and CMake fetches it like our other libraries.

5:45 To send Consul our requests, we need an HTTP client, and we're going to use cpp-httplib, by Yuji Hirose. It's a single header too, with a server and a client in it. Its README warns that its sockets block, which matters for a game server, as a request that waits on the network would hold up our tick, so we're going to keep its requests off the tick's thread. A client takes the host, and a request such as get returns a result, with the status and the body. Consul is going to sit next to our servers on Docker's network, so we use plain HTTP, and leave TLS out.

6:22 In the root CMakeLists, JSON comes first, from the archive that nlohmann's releases include. It holds the headers and the CMake files, without the tests and their data, so it's a much smaller download than the whole repository, and FetchContent makes it available like our other libraries. Then httplib, and before we fetch it, a few of its options. It would use OpenSSL if it found it, for HTTPS, so we switch that off, and the comment says why: Consul speaks plain HTTP inside Docker's network, so httplib needs no TLS or compression. Zlib, Brotli and zstd would let it read bodies that come compressed, and Consul's answers are small and plain, so those go too. No exceptions, as our style guide doesn't use them, so httplib leaves out the try and catch it has in places. And no install, as httplib goes into our own programs and nowhere else. Then FetchContent declares httplib, from the archive of its release tag on GitHub, and makes it available. Its target carries the no exceptions option as a definition, so that each file that includes httplib's header compiles it with that setting. In the server's CMakeLists, consul dot c c joins the server's core library, which now links httplib and JSON too, so that the world server, the login server and the tests can all use it.

7:49 Now the client for Consul, in a header of its own, consul dot h. After the header guard, like in our other headers, we include c s t d int, for the fixed-size integers of the ports, string, and utility for std move, then Abseil's status and status or, for what our functions return. World registration is what a world tells Consul: its name, the public host that players reach it at, its port, the port its metrics are on, and how many players it has right now. The Consul class holds the host Consul runs on, and k port is eight five hundred, Consul's HTTP port. The constructor is explicit, and takes the host by value, so that a caller's string can be moved in. Register sends a world's registration, and it's const, like the rest of the class, as the client keeps nothing between requests but the host. Its comment says that it registers the world, or registers it again with its players. Consul keeps the players in the registration, so sending it again is how they get updated. The comment goes on: its health check starts as passing, and keeps its status when it's registered again. Pass is what the heartbeat is going to call each two seconds, and as its comment says, it passes the world's health check, which turns critical ten seconds after the last pass. It returns not found when Consul doesn't know the world. Deregister takes a world out of Consul, by its name. Own address works out the address Consul reaches us at, and put sends Consul a request with a body, and both are private, as nothing outside the class needs them, while host holds Consul's host. With httplib and JSON kept behind this class, the world server and the login server see our own types and Abseil's statuses, and the rest stays in consul dot c c. In consul dot c c, after our own header, httplib's comes in angle brackets, as a single file at the top of its folder, then string and system error. From the libraries, str cat, Asio's connect, io context and UDP, and nlohmann's json. In the unnamed namespace, k service is the name that all our worlds register under, nanommo dash world, as Consul groups the instances of a service by its name. Connect makes an httplib client for Consul's port, with a second to connect, and a second to read the answer, as a world's next heartbeat is two seconds away. Answer turns httplib's result into a status, or the body. With no answer at all, that's unavailable, with the error httplib gives us, as a string. Consul answers four oh four about a world it doesn't know, so that's not found. Any other status but two hundred is an internal error, with Consul's body, which says what was wrong, and otherwise, answer returns the body. All our requests go through answer, so that they report errors alike. Register starts by working out our own address, on each call, so that a world that couldn't find Consul when it started finds it on a later beat. If there's no address, register returns the error, which the heartbeat logs. Then the registration, as a JSON object written as nested braces, which nlohmann turns into JSON. ID is the world's name, which tells the instances apart, and Name is k service. Address and port are the world's on Compose's network, where Prometheus is going to reach it, with the metrics port from meta, while a TTL check means that Consul doesn't connect to the world at all. Meta holds anything else we want, as strings, so the players, and the metrics port. Tagged addresses gives a service more addresses, each with a tag, and Consul's tag for the address that's seen from outside is wan, so the public host goes there, for the players. And the check: a TTL of ten seconds, which starts as passing, and Consul deregisters a world that stays critical for a minute. Put sends all that to the register endpoint, as a string, from dump. Pass puts to the pass endpoint of the world's check, which is service, a colon, and the world's name, the ID Consul gives a check that comes with a service. Deregister puts to the deregister endpoint, with the world's name on the end. Own address needs a bit more thought, as a container can have several addresses, and the world can't tell which of them Consul can reach. Over it, the comment says how we find out: the address Consul sees us at is the one the kernel picks to send to it from, and connecting a UDP socket sends nothing. So own address makes an io context of its own, as it runs on Consul's thread, away from the server's, with a resolver, a UDP socket, and an error code for Asio's calls to fill in, as without one, they'd throw. The resolver turns Consul's host and port into addresses, and it takes the port as text, so str cat writes it out. Then connect tries the addresses until one works, and as UDP has no handshake, connecting fixes where the datagrams go, and the kernel picks the address to send them from. Local endpoint reads that address back, and if a step fails, the error says we couldn't find Consul, with Asio's message. Otherwise, it returns the address that the socket got, written out as text. Last, put, which register, pass and deregister go through, connects, and sends the body to the path in an HTTP put, with a content type of application slash json. It returns the status of answer, and drops the body, as none of these three requests gets back anything that we need.

13:17 In server dot h, the server gets players, which returns how many players are in its world, as an int, for the registration. Then the world server's main, where most of today's server code goes. At the top, algorithm, atomic, for a flag that two threads share, functional for std function, optional and string view, and from Abseil, ascii, to check the world's name. From Asio, post, steady timer and thread pool, for the heartbeat, and our consul header. We checked the port inline, and now a second port needs that check too, so parse port goes into the unnamed namespace. It returns the port, or nothing when the text isn't a number from one to sixty-five thousand five hundred and thirty-five, so that the caller decides what to say about it. Simple a to i reads the text into an int, and the range check comes after it, before the cast. It takes a string view, so that it can read the argument, a C string, and a setting, a std string, without copying either of them. The world's port is still the first argument, with seven seven seven seven by default. Where main used to read an int and check its range on the spot, it calls parse port, and the error for a bad port stays as it was. The metrics port becomes a setting, nanommo metrics port, nine four six four by default, which parse port reads. Two worlds on one machine can't serve their metrics on one port, so each needs a metrics port of its own, and it goes into the registration, for Prometheus. A bad one stops the server, with what it should be, like a bad port does, so that a typo in the setting shows up straight away, and not later, as metrics that Prometheus can't find. The world's name comes from nanommo world, with Ember by default, as that's the world we've played in so far. It can't be empty or longer than sixteen characters, and all of checks each character with Abseil's ascii is al num, so that it's letters and digits. The name shows up in the client's list of worlds, and it goes into Consul's URLs, where a slash or a space would get in the way. The exposer serves the metrics on the port from that setting now. Further down, past MySQL and Redis, which don't change, create takes the port as it is now, without a cast, as parse port gives us the right type, along with the metrics, the tokens and the players, like before. The first log line starts with the world's name, so that you can tell two worlds' logs apart, and then the port it's listening on. The second log line says which port the metrics are on, as that's a setting now. Then Consul, on the host in nanommo consul host, and the world's registration, with designated initialisers, which name each field as we set it. The registration gets the world's name, the public host from nanommo public host, localhost by default, its port and the metrics port. The players are set on each heartbeat, as they change. Consul gets a thread of its own, from Asio's thread pool, with one thread in it, and the comment says why: so that a slow answer can't hold up a tick, as httplib blocks until it has one. Waiting is an atomic bool, which both threads can read and write safely, and it's true while a beat waits on Consul. As its comment says, a beat that's still waiting on Consul is enough, so that beats can't pile up behind a slow Consul. Listed is the number of players Consul has for us, if it has us at all, and said is the last line we logged. The heartbeat is a timer on the io context that runs our ticks. If the tick loop gets stuck, the heartbeats stop with it, and Consul marks the world critical. Beat is a std function, so that its body can hand beat to async wait again, like the saver's timer in episode thirteen. On each beat, the registration gets the number of players, on the tick's thread, where reading the world is safe. Exchange sets waiting to true and returns what it was, in one step, so unless a beat is already waiting, post hands a lambda to Consul's thread. The lambda captures Consul, listed, said and waiting by reference, as main joins Consul's thread before they go away, and the registration by value, since the next beat changes it on the tick's thread. The lambda's status starts as ok, for a beat that has nothing to register. There, the world registers again when its players have changed since Consul last heard of them, and when Consul hasn't heard of them at all, as listed is empty then. Otherwise the beat skips registering, as the registration would be no different. Then the beat passes the check, and the comment over it says why each beat does: registering again keeps a critical check critical, so without a pass, a world that went critical once would stay out of the list for good. When it all worked, listed is the players Consul has. After an error, listed goes back to nothing, and the comment says why: Consul forgets its worlds when it restarts, and a pass for a world it doesn't know is not found, so the next beat registers ours again. News is what we'd log: that we registered, under the world's name, or the error, as a string from the status. Said keeps what we logged last, so the log gets a line when the news changes, and the comment over it reads: once when it changes, not with each heartbeat. Last, waiting goes back to false, so that the next beat can go to Consul. The timer waits two seconds, and beats again, while the first beat runs straight away. When the world stops, once io run returns, join waits for a beat that's still on its way, and deregister takes the world out of Consul, so that it leaves the list at once.

18:49 In compose dot yaml, a consul service, from HashiCorp's official image. Its command runs Consul's agent in development mode, dash dev, which is one server that keeps everything in memory. That suits us, as each world registers again within a few seconds after Consul restarts. Development mode logs a lot, so its log level goes down to warnings. Port eight five hundred is published, for Consul's UI, and its health check runs consul members, which works once the agent is up. The server gets nanommo consul host, set to consul, the service's name on Compose's network, and it waits for Consul to be healthy. In compose dot prod dot yaml, Consul's port goes on the server's localhost, like Prometheus's and Grafana's, as anyone who reached Consul could register a world of their own.

19:40 Let's build it, and start the world server, with Consul and the rest of what it depends on, and the build is sped up. Ember is listening, but its log says that it couldn't reach Consul, as httplib couldn't establish a connection. And yet, Consul is up, and its health check passes, as compose waited for it.

20:00 Consul's health check runs inside Consul's own container, while the world connects from another one. Consul's agent listens on one twenty-seven dot zero dot zero dot one unless it's told otherwise, which is the loopback of its own container, so its health check gets an answer, and other containers get nothing. The official image tells it otherwise in its own command, which is agent, dash dev, dash client, zero dot zero dot zero dot zero, and a command in compose replaces the image's whole command, flags and all. So when we added the log level, we dropped dash client. Dash client goes back in, set to zero dot zero dot zero dot zero, which makes Consul listen on all of its container's addresses. A flag lost like that is easy to miss, as the image's command isn't anywhere in our file.

20:47 Let's start it again, with Consul's new command, and this time, the world's log says that it registered with Consul as Ember.

20:58 Consul's UI is on port eight five hundred, and under the nanommo world service, Ember is the one instance, with its checks passing. Ember's page shows the address it registered, on Compose's network. Its service check is a TTL check, which keeps passing as long as the heartbeats come. And under tags and meta, there's the number of players, none so far, and the metrics port.

21:22 Now let's see what happens when a world goes away. Docker compose stop sends the world SIGTERM, so it deregisters on its way out, and Consul's health endpoint, which curl asks and jq reads, has no worlds left in it. We start the world again, and this time we kill it, which sends SIGKILL, like a crash would, so it has no chance to deregister. Right after, Ember is still there, and passing, as its last heartbeat is less than ten seconds old. Twelve seconds later, its check is critical, so a list of the worlds that pass leaves it out, and a minute later, Consul deregisters it. If it starts again within that minute, its first heartbeat passes its check, and Ember is back in the list.

The list of worlds

22:08 Now the login server is going to list the worlds, so in nanommo dot proto, the login request gets list worlds, for a request that asks for the worlds and nothing else. The list of worlds is no secret, much like a game's server status page, so the client can ask for it before anyone logs in. A world listing is what a player needs to know about a world: its name, the host and port to connect to, and how many players it has. And the login reply gets the worlds, as a repeated field. In consul dot h, we include vector and our protocol's header, and worlds returns the worlds that pass their health checks, by name, with the host and port that players reach them at. Get sends a request that reads from Consul, and returns the body. In consul dot c c, algorithm, for sort, c s t d int, vector, and Abseil's numbers, to read the players. Find is a template, like the one in episode ten, for a type T that we name when we call it. We need it as our style guide doesn't use exceptions, and the comment over it says why that matters here. Nlohmann's json throws when a value isn't the type we ask for, so this checks first, and returns a null pointer when it isn't there or isn't a T. Find takes a JSON value and a path, which becomes a JSON pointer. Contains tells us whether the path exists, so that at doesn't throw, and get pointer returns null when the type doesn't match, where get would throw. Consul's catalog lists each world that registered, whether its check passes or not, so worlds asks for the health of nanommo world instead, and passing, on the end, makes Consul leave out the worlds that are critical. Parse gets false for allow exceptions, so a reply that isn't JSON comes back discarded, and is array is false for that, which gives us an error. Worlds starts with an empty vector of listings, and goes through the entries, one for each world that passes. Then, for each entry, we find what we registered, by its path, which starts at the service. Name is the service's ID, which is the world's name. Host is under tagged addresses, in wan, which is the address that we registered for the players. Port is next to it, and it's a number, which nlohmann keeps as an unsigned sixty-four bit integer, so that's the type we ask find for. Players is under meta, which Consul keeps as a string, so simple a to i reads it into count, which is thirty-two bits, like the field in our listing. If any of them is missing, the port isn't one a client could connect to, or players isn't a number, that entry stays out. Anything on Compose's network can register under our name, so the comment over it says that a stray entry like that stays out, and the other worlds still make the list. Otherwise emplace back adds a listing to the end of worlds, and returns it, so we can fill it in: the name, the host, the port, cast down to thirty-two bits, and the players. Then the worlds get sorted by name, with name as a projection, so the list doesn't move around from one refresh to the next. Last, get connects and returns answer, which is Consul's body when it worked, so that worlds can parse it.

25:12 The login server gets Consul, so that it can answer a request for the worlds. Its header includes chrono and ours, and create and the constructor take a Consul, which the server keeps a reference to, as main owns it. List worlds is a private function that answers a request like that, with Consul's list. Next to Consul, the server keeps the last list it got, as a reply that's ready to send, and when it got it. In login server dot c c, chrono and vector, and create passes Consul on, and the constructor keeps it, next to the members it had. A request comes in through the session's on message function, which parses it, as before. When a request asks for the worlds, the login server sends the reply from list worlds, framed like any message, and returns, before it gets to a password. List worlds starts with the time, from the steady clock, which doesn't jump when the system's clock is set. Over it, the comment says why it needs the time: a list from the last second is new enough, and a crowd of clients costs Consul a request a second, while this thread waits for its answer. So when the last list worked, and it's less than a second old, list worlds returns it. This thread handles the logins too, and they wait with it, so the less often it asks Consul, the better. Otherwise, it asks Consul for the worlds, which come back as a vector of listings, or an error. If Consul can't answer, it logs why, on standard error, and the reply says that the server couldn't list the worlds, and to try again later, as a player can't do anything about Consul. Otherwise the reply is ok, and assign copies the worlds into the reply's repeated field. The reply and the time become the last list, for the next second's requests. In the login server's main, our consul header, and then Consul, on the host in nanommo consul host, which goes to create, after the tokens. In compose dot yaml, the login server gets the Consul host too, and waits for Consul, and so do the tests, so that a test can talk to Consul.

27:18 The test goes against a real Consul, like our MySQL and Redis tests do. After our consul header and GoogleTest's, it includes algorithm, for find, c s t d lib, for get env, string and vector, and our header for test services, for test name. A using declaration means that world listing doesn't need its namespaces each time. Lists a world until it deregisters skips unless nanommo consul host is set, so that it runs in Docker and in CI, next to Consul, and not on a machine without one. It makes our Consul client on that host, and a random test name, so that two runs can't clash, then registers a world under that name, with a public host of example dot test, port seven thousand, and twelve players. Then the list of worlds from Consul has to include it. Find takes the name and a projection, so it compares each listing's name, and assert not equal stops the test if it isn't there, as the lines after it read the listing. The host, the port and the players are what it registered, so a player would connect to the right place. Then it deregisters, and the next list of worlds leaves it out. Consul copies a registration into its catalog before it answers, so the test doesn't need to wait. The test joins the server's tests in CMake, next to the others.

28:36 Compose builds the tests, and runs them next to MySQL, Redis and Consul, here the one for Consul, which passes.

Choosing a world

28:52 On to the client, where the login screen gets a page before the login, with the worlds. In login screen dot h, vector, and the comment over build says what it does now. It draws the worlds, or the login for the world picked, and returns what to ask the login server, if anything: the worlds when they're shown and refreshed, or a login. Show worlds hands the screen a list, and world is the world picked, if one is. Privately, build worlds and build login draw a page each, and return what that page asks for. The screen keeps the list of worlds, the index of the one that's selected, and the world picked, as a copy of its listing, which stays empty until you pick one. In login screen dot c c, algorithm, string, utility and vector, and str cat. Both pages use the window flags that the login had: the window resizes to fit, it can't be collapsed, and it doesn't move. So they go in a constant, k window flags, and the red line for an error goes in show error text, with its warning icon, as both pages show errors. Build still puts the next window in the middle of the screen, which suits both pages, and then it draws the login when a world has been picked, and the worlds otherwise. Show worlds takes the list by value, and moves it in, like Consul's constructor does with its host. Clamp keeps the selection inside the list, in case a refresh brought fewer worlds. Build worlds begins a window called Worlds, with our window flags, and one more on top of them, no nav. ImGui moves between widgets with the keyboard, which the login form uses, with Tab. The comment over this line says why the worlds page turns that off: the arrow keys and Enter pick a world, so ImGui's own keyboard navigation stays out of this window. The page asks for the worlds when the window appears, and when F5 is pressed, like reloading a page in a browser. Down and up move the selection, within the list, so down stops at the last world, and up at the first, and Enter picks the world that's selected. Pick is true when Enter was pressed in this frame, and a click on continue, further down, sets it too. For each world, push ID gives its row an ID of its own, from its index, as ImGui tells widgets apart by their IDs, which otherwise come from their labels. Each row is a selectable, ImGui's widget for a row that can be selected, with the world's name on it, and a click selects it. It's a hundred and sixty pixels wide, so that what comes after it lines up from one row to the next. On that line, after an icon of people, goes how many players the world has, in grey, and pop ID ends the row's ID. With no worlds, the page says that no worlds are up. Then a refresh button, with a circle of arrows on it, which asks for the list again, like F5 does. On that line goes continue, which picks the selected world, like Enter, and under the buttons, the error, if there is one. A world that's picked becomes world, a copy of its listing, and the error goes, so that the next frame draws its login. Without a refresh, build worlds has nothing to ask the login server, so it returns nothing, and with one, it returns a request for the worlds, for main to send. Build login is the login window we had, and its title says log in to and the world's name, with str cat, so you can see which world you're logging in to. It gets a third button, worlds, with an arrow that points left. When it's pressed, the world picked is reset, and so is the error, so the next frame draws the list, which asks for the worlds again as it appears, in case one came up or went down while you were on the login.

32:28 In the client's main, k game port goes, as each world has a port of its own now. List worlds sends the request for the worlds with log in, the function we log in with, which sends any request to the login server and waits for its reply. If it can't reach the server, the screen shows that it couldn't, with the status's message, and if the reply has an error, the screen shows that error. Otherwise the screen gets the worlds, copied from the reply's repeated field into a vector. Enter takes the world that was picked, and the log says which world we're entering, as with a few worlds up, the log is where you'd look to see which one a client joined. The client connects to the world's host and port, from its registration in Consul, with the port cast to sixteen bits, as protobuf has no integer smaller than thirty-two. The client checks the world's certificate against that host, like it has since episode fourteen, so a world needs a certificate with its public host on it. In the loop, a request for the worlds goes to list worlds, and the list shows up on the next frame. A login goes to enter, with the world picked, and the client's argument is the login server's host now, while the rest comes from the list.

33:40 Let's run it, with the login server and one world, and the client's terminal under its window. The client opens on the worlds, and Ember is the one that's up, with nobody in it. Enter picks it, and the login window says that it's for Ember. I create my account, and I'm in Ember, on the island we know, and the client's log says which world it entered.

34:05 In compose dot yaml, the server's public host comes from the shell, with localhost by default, so that you can start the worlds with your machine's address on your network, and play from another machine. On a server, players reach the worlds at the server's address, so in compose dot prod dot yaml, the world's public host comes from dot env, like the password. Deploy dot s h has that address, as it deploys to user at address, so the public host is what comes after the at, and the comment says so. SSH joins its arguments into the command it runs on the server, so putting public host first sets it for the script. Unless dot env has it already, the address goes in, before the build, as the production compose file won't build without it. That covers a server we deployed to before today, so the next deploy registers the worlds with the server's address.

A second world

34:59 Now let's run a second world in Docker, next to Ember. In compose dot yaml, the server, which is Ember, gets its name and seed written out, so that the file says which world it runs. Then frost, which extends server, so it takes its build, its environment, its volumes and its dependencies from it. Its command is the port, seven seven eight seven, which goes to the image's entry point, the world server, as its first argument. Its environment names it Frost, with seed two, for an island of its own, and its ports override the server's, so it publishes seven seven eight seven, for TCP and UDP. Its metrics stay on port nine four six four inside its own container, where Prometheus reaches them. In compose dot prod dot yaml, frost needs settings of its own, as extends works within a file, so the production settings for the server don't reach frost on their own. So frost gets the password and the public host from dot env, and restarts like the rest.

35:56 Prometheus scraped server colon nine four six four, which is Ember. Instead, it's going to find the worlds in Consul, with consul SD configs, which asks Consul for the instances of nanommo world, each five seconds. Relabel configs turn what Consul tells Prometheus into a target. The address and the metrics port from meta, joined by a colon, become the address to scrape, and the service's ID, the world's name, becomes a label, world, on all of that world's metrics. In the dashboard, the panels about the worlds show them by that label, so the players online, the ticks and the bytes sent are per world. The tick durations and the saving times are histograms, so their buckets are summed by world, for a line for each world. And load dot s h measures Ember, with a matcher on that label, world equals Ember, in each of its queries. Over it, the comment says why: the bots play in Ember, so that's the world we measure. Without the matcher, Frost's quiet ticks would get mixed into the tick durations, and the players online could come from either world.

37:01 Let's start both worlds, along with Prometheus, and open its targets page. It lists the two worlds that it found in Consul, each with its world label, Ember, and Frost, at their addresses on Compose's network, and both are up.

37:16 Now let's play in both worlds, with a client for each. On the left, I pick Ember, and create my account, then I run a little way from the middle, and the character screen shows where I am. On the right, a second client opens, and I pick Frost, and log in to my account there. Its character screen shows the position from Ember, to a tenth of a metre. Both worlds load and save a single row for each account, so my character in Frost is the one from Ember, standing where it stands in Ember, on an island where that could be the sea. And whatever I did in one world would move my character in the other, so each world needs players of its own.

Each world's own players

37:58 In MySQL, the players table gets the world as part of its key, so an account has a row for each world it has played in. In Redis, a player's hash gets the world in its key, as player, the world and the account. The hash also says which world and account it's for, so that the saver, which takes the players of all the worlds to MySQL, doesn't have to take keys apart. Our players so far have played in one world, so a migration puts them in Ember. The accounts stay as they are, shared, so that one login works in each world.

38:30 The migration is V three, players per world, after the two from episode eighteen. It adds the world column, of up to sixteen characters, like the names the servers take, after the account ID, with Ember as its default, which fills it in for the rows we have. The primary key was the account ID, and it becomes the account and the world together. The foreign key to the accounts table needs an index that starts with the account ID, and the new key starts with it, so MySQL lets the old one go. The default stays, as it matters during a deploy, while the servers from before are still running. Flyway migrates before compose starts the new servers, and the old ones save their players without a world, which lands them in Ember, where they belong. Without the default, those saves would fail, and the players would lose their last moves.

39:18 In players dot h, string, and load takes the world, so its comment says it loads the account's player in that world. Save takes the world too, while flush, which the saver calls, takes the unsaved players of all the worlds to MySQL, as its comment says now. In players dot c c, key takes the world and the account ID, and makes player, the world and the account, with colons between them, like Redis keys usually are. Load joins the players table on the account and the world, so in a new world, the account's player starts in the middle, with no experience, from coalesce's defaults. The world is the first value for the query, before the account ID, in the order of the question marks. Then load reads the Redis hash with the world in its key. Save gets that parameter too, for the players that this server changed, which come in as before. For each of them, H set writes to the key with the world in it, and it puts the world and the account into the player's hash too, next to the progress. Then it marks the hash's key unsaved, where it used to mark the account ID. So flush pops keys out of the unsaved set, and for each key, it reads the hash's world and account, along with its progress. As the comment says, before worlds, the set held account IDs, for hashes with no world, and a deploy can leave some behind, from the servers before it. For one of those, flush reads the hash at player and the ID, which has no world or account in it. The insert takes the world as its first column, and each row gets six question marks now, with null if still around the last one, for a hash that has no health. An account ID from before worlds gets Ember and the ID in front of its values, and the rest go in as they come out of each hash, world and account first, and health last, like before. In server dot h, create takes the world's name, after the interval, by value, as the server stores it. Its comment says that the name keeps the world's players apart from other worlds', as one account can have a player in each world, while the seed makes the terrain, like before. The constructor takes the name too, and the server keeps it in name, next to the seed, as it loads and saves its players with it for as long as the world runs. In server dot c c, create's parameters change like the header's. We could have given the name to players instead, when main makes it, though the saver uses players too, to flush all the worlds at once, so load and save take the world, and the server passes its name. Create moves the name into the constructor, which moves it into name, so the string isn't copied on its way in. Handle hello loads the player with the world's name, and save changed saves the players with it. And in main, the name goes to create, between the interval and the seed.

42:02 On the website, the rankings had a row for each account, and now they have one for each account and world, so the query gets the world too. The table gets a world column, and the line that says nobody has played yet spans five columns now.

42:17 Let's see the migration on a database from before this episode, with a player in it, which I set up off camera. Starting both worlds runs Flyway first, and its log says that it migrated the schema to version three, players per world. The player who was there before is in Ember now, with their position and experience as they were.

42:38 Let's play in both worlds again, with the migration in. In Ember, on the left, I create my account, and run a little way from the middle. On the right, I log in to Frost, with that account. My character in Frost is a new one, in the middle of Frost's island, with no experience, while the one in Ember is where I left it, so each world has players of its own now.

43:03 In the players tests, tear down deletes three keys now, with a loop over a braced list of them: the account's player in Ember, in Frost, and the hash from before worlds. Del removes each of them from Redis, one key at a time. Then S rem takes the two keys and the bare account ID out of the unsaved set, so that a test leaves nothing behind for the next run. Each load and each save in the tests names a world now. Loads where the player was saved becomes loads where the player was saved in that world. Its saves and its load name Ember, and the player comes back where it was saved, like before. At its end, the account's player in Frost is in the middle, with no experience, as each world keeps its own. A separate player in each world is the main change today, so this test pins it down. Flushes the players to MySQL saves a player in Frost, with save. Then it needs the account ID as a string, for the commands that follow, and over it goes a comment on what comes next: a player saved before worlds, an account ID in the set, and its hash. A pipeline puts the two of them into Redis by hand, with H set for the hash, and S add for the set. After flush, the number it saved counts at least those two players. Then one query reads the account's rows, ordered by world, and there are two of them: the one from before worlds, in Ember, with the values from its hash, and the one in Frost, each with its own progress. Expect equal compares them with the rows we expect, written as a vector of rows, each a vector of strings. Comparing the whole result at once means that a missing row, or one too many, fails the test, like a wrong value does.

44:38 Then all the server's tests, which compose builds first, and runs next to MySQL, Redis and Consul, and they pass.

A third world

44:49 So far, our second world came with a change to compose dot yaml, though a world doesn't need one. So let's start a third world from a terminal, with the client on its list of two. Gale, with seed three, on port seven seven nine seven, with a metrics port of its own, as it shares this machine with the other two worlds. Its log, along the bottom, says that it registered with Consul. F5 asks for the list again, and Gale is in it. I pick Gale, and create my account, as this database starts empty, and I'm on Gale's island, with nothing changed in the client, in the login server, or in a file.

45:28 In Docker, a third world is a docker compose run of the server's service, with the world's name and seed, its port as the argument, and that port published, without a line written into compose dot yaml. Consul's list of the worlds that pass has Gale in it, with the public host and port that players reach it at. And Prometheus, which asks Consul each five seconds, scrapes Gale too, with its world label, so our dashboard shows three worlds without a change to Prometheus's configuration.

Docs

45:57 Last, the README changes in a few places, starting with the build section, which lists the two new libraries, and says that Docker Compose runs Consul too, before the login server and the worlds. The paragraph on the client says that it opens on the worlds that are up, how to pick one with the keyboard or the mouse, and that F5 refreshes the list. The section on accounts says that the token goes to the world you picked, and that the tests need nanommo consul host, like the hosts for MySQL and Redis. The section on saving players gives the keys with the world in them, and the hash's world and account, and monitoring says that Prometheus finds the worlds in Consul, with their names as a label. Then a new section on worlds: a table of the settings a world takes, how each one registers with Consul and passes its check with a heartbeat from its tick loop, and how to start a third world without changing a file. It also says that a world's name is its ID, so two worlds with one name would share their players, and the second to register would replace the first in the list. The bots play in Ember, as they connect to its port. Deployment says which ports the worlds are on, that their public host comes from dot env, which deploy dot s h fills in, and that Consul listens on the server's localhost, like Prometheus and Grafana. And restarting the servers after new certificates names frost now, next to the login server and Ember.

Wrap-up

47:23 So, today Consul joined our servers, as the registry of the worlds that are up. Each world sends Consul a heartbeat each two seconds, from its tick loop, on a thread of its own, so a world that stops or crashes drops out of the list, and comes back when it ticks again. The login server lists the worlds that pass their checks, with nlohmann's json to read Consul's answers, and the client picks a world before it logs in, while Prometheus finds the worlds in Consul too. Each world keeps its own players, with the world in the players table's key, and a third world joined the list and our monitoring with a single command.

Playing it

48:55 Let me know in the comments what you'd like to see next. And if you want to follow along with the series, subscribe to the channel. I hope you find this useful, and thanks for watching.