Multiplayer with protobuf and Asio

Intro

0:00 Hello and welcome back to episode three, where we're going to write the server, and get several players moving in one world. Let's begin with a quick look at where we're going to be by the end of the episode.

0:14 These three clients share one world, connected through our server. Each one sends its keystrokes, and the server processes the intent and broadcasts to the rest, which needs networking and packet management.

0:26 On the website of the series you can find more information about the project. The code for the project is open source on GitHub, tagged at each step of each episode so that you can follow along.

Last time

0:38 Now, before we get to today's work, let's quickly go over what we did last time. In the last episode, we started the client, with a window from raylib and a menu from Dear ImGui. Its tests render frames into a texture, and in Docker, they run on a virtual display, without a screen or a GPU. Then we wrote our first shaders, and drew a cube as the player, on a ground of tiles. And the keys move it, at four tiles a second. However, the client moves the cube on its own, and nobody else knows where it is.

Today

1:12 Today, we start writing the server, which is going to run the world for all the clients. The client and the server agree on their messages through protobuf, and they send them to each other over TCP. And by the end, we're going to have several clients connected to one server, each moving its own cube, and seeing the others move too. All right, let's get started with why the server should be the one deciding where everyone is.

The server decides

1:36 So far, the client decides where its cube is, and in an MMO, that can't stay that way. If each client decided where its own player stands, a modified client could put its player anywhere it likes, and the other clients would have no way to tell. So the server is the one place that decides what happens in the world, and the clients show it. That's called an authoritative server, and it goes with the thin client I talked about in the first episode. Instead of a position, a client sends its input, which is the way its player wants to move. The server moves all the players on each tick, twenty times a second, with the tick interval from episode one. After each tick, it sends each client the world, which for now is a list of the players and where they stand. And the client draws whatever world it gets, without moving anything on its own. So when you press a key now, your cube moves once your input has reached the server, and the world has come back to you. On one machine or a local network, that's a few milliseconds. Over the internet, it can be a lot more, and games hide that delay with tricks such as prediction, where the client moves your player straight away and corrects it later.

TCP, for now

2:44 For the client and the server to talk, we have to pick between the two ways the internet gives us to send data, TCP and UDP. TCP gives us a connection, and a stream of bytes that arrives reliably and in order. If a packet gets lost on the way, TCP sends it again, and the operating system takes care of all of that for us. The catch is that the bytes after a lost packet wait until it's been resent, even when they've arrived already. That's called head-of-line blocking, as the lost packet at the head of the line holds up everything behind it. For position updates, that's a bad trade, as by the time a lost update is resent, a newer one is on its way anyway. UDP sends separate packets instead, with none of those guarantees about delivery or order. A lost packet is gone, and the next one carries a newer position. That's why fast-paced games, such as shooters, tend to use UDP. However, then our own code has to make sure that the messages that do matter, such as a chat line or a trade, arrive, and arrive in order. That's a fair bit of work, and there are whole libraries that do it on top of UDP. So, my decision is to start with TCP, as in the kind of MMO we're building, you pick a target and your character fights it. So a short hiccup now and then doesn't ruin anything. World of Warcraft, for example, has run on TCP for years. TCP also lets us work on the contract and the syncing first, instead of on lost packets. And starting with TCP doesn't lock us in, as the library we're going to use, Asio, does UDP as well. And the rest of our code doesn't depend on which transport carries its messages. We're going to come back to this once we can measure it, with a load test and some packet loss simulated in Docker. And if UDP turns out to be the better choice, we'll change our mind then.

A contract with protobuf

4:32 Before sending anything, the client and the server have to agree on what their messages look like, and that's the contract between them. The quickest way would be to send a C++ struct's bytes as they sit in memory. However, that breaks as soon as the two sides differ. The order of the bytes in a number depends on the processor, and the compiler can add padding between fields. And adding a field moves the ones after it, so an older client reads garbage. JSON is the usual answer on the web, and it's readable, which is nice for debugging. However, it's text, so it's bigger and slower to parse, and nothing checks that both sides agree on its fields. So we're going to use protobuf, Google's protocol buffers, which I proposed in episode one. We describe the messages once, and a compiler generates the code that reads and writes them. It does that for C++ and for a lot of other languages, so later on, a load-testing bot in Python could speak our protocol without us writing it twice. Its bytes are compact, and each field has a number, which is what lets us add fields later without breaking the clients that don't know about them yet.

5:37 Protobuf's documentation lives at protobuf dot dev, and its overview page is a good place to start. The page describes protobuf as a language-neutral and platform-neutral way to serialise structured data, which means turning it into bytes and back. A message lists its fields, each with a type, a name and a number. protoc, the protocol buffer compiler, turns each message into a class in our language. In C++, that's a getter and a setter for each field, and functions to turn the message into bytes and back. The numbers are what goes into the bytes, not the names. That's why we can rename a field without breaking anything. However, a field's number mustn't change once it's in use.

6:20 Both the client and the server are going to use the contract, so it gets a folder of its own, protocol, next to the two of them. The file is nanommo dot proto, named after the game, with the extension protobuf's files use. Its first line says which version of protobuf's language we write it in, proto three. The documentation's examples use editions, which are protobuf's newer way of choosing its features. However, proto three is what most code out there is written in, and what most examples online assume, so it's what we start with. Moving to editions later on should be a small change. Then the package, nanommo dot protocol, which keeps our message names apart from anyone else's. In C++, the package becomes the namespace nanommo protocol, and that's going to matter to us. The server is going to have a World class of its own, and the two mustn't clash. Now the messages, starting with Input, which goes from a client to the server. A client sends it whenever the keys it holds change. It carries the direction its player wants to move in, and not a position, so the server stays in charge of where everyone is. It has two floats, x and z, each from minus one to one, and there's no y, as nobody jumps or flies yet. The equals one and equals two after the fields aren't default values, and they're the field numbers we saw in the docs. Then Player, which is how the server describes one player to the clients. It has an id, as a u int thirty two, which is an unsigned thirty-two-bit number. That's room for about four billion players before the ids wrap around. And it has where the player stands on the ground, as an x and a z, again with no height. And World is what the server sends on each tick. You is the id of the player that's yours, so that a client can tell its own cube apart from the others. And repeated Player players is how protobuf says a list, which here holds all the players in the world. I want to keep the contract this small to begin with, one message up and one down, so that when something goes wrong, there's very little to look through. Next is the top-level CMakeLists, where we fetch the protobuf library with FetchContent, like we did with GoogleTest and raylib before it. Protobuf is built on abseil, which we've fetched since episode one, for our statuses. Abseil's block sits at the top of this file, so by the time CMake gets to protobuf, abseil's targets are already there. So protobuf finds them and builds on them, and without them, it would try to find abseil on the system. On my Mac, that's an older one from Homebrew, and the two don't build together. Then we set two of protobuf's options, ahead of protobuf, so its build sees them. Protobuf install is off, as we use protobuf from our build folder, without installing it anywhere. And protobuf with zlib is off too, as zlib is for compressed streams and we don't use that. Then protobuf, at version thirty-six point two, marked SYSTEM like raylib, so the compiler doesn't warn us about code that isn't ours. Make available then builds protobuf, along with protoc, the compiler, from source. We could install protoc with Homebrew or apt instead, and the first build would be a lot quicker. However, the protoc that generates our code has to match the library it's compiled against. A protoc from the system tends to be a different version from the protobuf we fetch. Built together, they match on whichever machine builds them, my Mac, Docker or CI. The price is the first build, which takes a few minutes longer. Next, we include protobuf generate dot cmake, from protobuf's source. It gives us protobuf generate, a CMake function that runs protoc for us, which we're going to use in a moment. And further down, add subdirectory protocol, above the server and the client, which are going to use it. The protocol folder gets a CMakeLists of its own, where add library makes our protocol library. For now, the library's single source is the proto file. CMake doesn't build proto files on its own, so we call protobuf generate. Given our target, it finds the proto file among the library's sources. It adds a step to the build that runs protoc on it. Import dirs is where protoc looks for proto definitions, and we give it the project's root. So protoc knows ours by its path from there, protocol slash nanommo dot proto. Protoc out dir is where it writes, and we give it the root of the build folder, so what it generates keeps that path. protoc writes protocol slash nanommo dot p b dot h, and nanommo dot p b dot c c next to it, in the build folder. Both are added to the library, and compiled like the rest of our code. Then set source files properties, as the generated source sets off some of the warnings we switched on in episode one. It's protoc's code, and not ours to fix, so that one file gets dash w, like ImGui's sources in episode two. Target include directories makes the build folder's root a public include directory. Public means that whatever links the library gets it too. So the server and the client include protocol slash nanommo dot p b dot h by its path from the root, like our own headers. The library links with protobuf's runtime, lib protobuf, which the generated code calls into. It's public, as code that uses our messages uses protobuf's classes too. Then the protocol gets tests of its own, like the server and the client have. Add executable makes the test program, which links with our library and gtest main, and gtest discover tests registers its tests with CTest. So ctest, and the Docker build, run them along with the others. The test file starts with string, then gtest, and then protocol slash nanommo dot p b dot h, the header protoc generates. That include already tests the build, as this file wouldn't compile if the generation step didn't run. The first test, world survives the round trip, sends a World through bytes and back. That's what's going to happen on each tick, between the server and each client. So we fill a World in, and set you to two. Add players adds an entry to the list, and hands back a pointer to it. Protobuf owns the new entry, inside the world, and the pointer lets us fill it in. So we set its id, its x and its z. One and a half and minus three are on purpose, so that a lost fraction or a lost sign would show up. Serialize as string turns the message into bytes, and parse from string reads them into a second World. It returns false when the bytes don't make sense, so we assert that it's true. If it isn't, the test stops right there, as there's no point checking the values of a message that didn't parse. We expect you to be two, and then we assert there's one player, before we look at it. That way, a missing player fails the test, instead of reading past the end of the list. And we expect its id, its x and its z to come back as we put them in. Expect equal on floats is fine here, as nothing is computed: the bytes carry each float as it is. The second test, zeroes are not sent, pins down how compact the format is. An Input with x set to one, and z left at zero, comes out as five bytes. One byte says which field comes next and what type it is, and four bytes hold the float. Z costs nothing, as zero is its default, and protobuf leaves out a field that holds its default. So when you let go of the keys, the Input a client sends is empty.

14:08 So, let's look at how this fits into the build. We edit one file, nanommo dot proto, and that's the file we commit. When we build, CMake first builds protoc, and then runs it on the proto file. The header and the source file it writes are generated by each build, so they don't go into git, and they can't go stale. Those compile into our protocol library, and both the server and the client link it, so they're built from one contract. Docker runs this whole build as part of its build stage. So it happens on Ubuntu with GCC, and in CI on GitHub's machines, without any extra steps. And when someone changes the contract, both sides get rebuilt. Code that no longer matches it fails to compile, instead of a client misreading the server's messages while the game runs.

15:00 Let's build it, to see protoc at work on our contract. I've built everything once already, as protobuf and protoc take a few minutes the first time, and I've touched nanommo dot proto, as if we'd just edited it. The first line, running C P P protocol buffer compiler on nanommo dot proto, is protoc at work. Then the compiler builds what it wrote, into our library, and the tests link with it. The two generated files sit in the protocol folder of the build. And the header has the setters we used in the test, such as set you, and set x, one for each field. Both of the protocol's tests pass, so a World survives the round trip, and zeroes aren't sent.

Framing messages

15:46 Protobuf turns a message into bytes, but it doesn't split the bytes into messages for us. TCP gives us a stream of bytes, not separate messages. If the server sends two worlds, they may arrive in one read, glued together. Or one of them may arrive split over two reads. And protobuf's bytes don't say where a message ends, as protobuf leaves that to whoever sends or stores them. The usual fix is called framing, where before each message, we write its size, in four bytes. The reader waits until it has those four bytes, reads the size, and then waits until it has that many more. The size goes most significant byte first, which is called network byte order, so that machines with different byte orders read it alike.

16:30 Framing goes into the protocol library, as both sides need it, in a header and a source file of its own. After the header guard, the header includes cstddef, for size t, which is the type C++ uses for sizes. And optional, string and string view, which we're going to need further down. Then abseil's status, which we saw in episode one, for when a stream turns out to be broken. And message lite dot h, from protobuf, as message lite is the base class of all protobuf messages. So a function that takes one works with Input, with World, and with any message we add later. First comes a limit, k max message size, at sixty-four kilobytes. It's constexpr, like the tick rate in episode one, so it's a constant the compiler knows while it compiles. Our worlds are a few dozen bytes for now, so sixty-four kilobytes leaves a lot of room. Above it, we write that anything bigger is garbage, as without a cap, we might buffer up to four gigabytes. The size in front is four bytes, so it can claim up to about four billion. A broken stream, or someone sending us junk on purpose, could make us wait for that much, and keep all of it in memory. Then frame, which takes any message, and returns what we write to the socket for it. It returns a string, as in C++, a string is a handy, growable buffer, and Asio can send one as it is. Its comment says the size goes in front, as four bytes in network byte order. Reading is harder than writing, as bytes arrive whenever the network delivers them. So reading gets a class, FrameReader, which holds on to what's arrived so far. Append adds bytes as they come in, however the stream happened to cut them up. It takes a string view, which looks at someone else's characters, without copying them. So the caller can pass us the part of its buffer that was filled. Next hands back the next whole message, if one has arrived in full. It returns an optional string, which is the standard library's way of saying a value that may not be there. Here, an empty optional means the message isn't complete yet, and the caller should wait for more bytes. And we document that after a message that's too large, it returns nothing more, and status says why. The guide doesn't use exceptions, so the reader keeps the error, as an abseil status, and whoever reads from it asks for it. Status is an accessor, like the renderer's background in episode two, so it's in lower case, const, and hands back a const reference. The bytes we've got so far live in buffer, a private string. And next to it is the status, which starts out OK. Keeping the error in the reader means each connection's reader carries its own, and a test can look at it too. The source file starts with its own header, as the guide asks. Then cstdint, which gives us the fixed-size integers, such as u int thirty two, and string, while Abseil's status, for the error we keep in the reader, comes with our header. In an unnamed namespace, which keeps it private to this file, k header size is four, the bytes the size takes up. Frame takes the message by const reference, so there's no copy, and it can't change it. It first serialises the message into a string, which is where protobuf does its part, and the rest of the function is ours. Then we take the body's size as a u int thirty two, as four bytes is what the header holds. Static cast is C++'s way of converting a value on purpose. Here, it goes from size t, which is sixty-four bits on our machines, down to thirty-two. Our messages stay well under the sixty-four kilobytes a reader accepts, so no bits are lost. Then we make a string for the frame, and reserve room for the header and the body up front. So the string allocates its memory once, instead of growing as we add to it. Then we push the size, a byte at a time. Shifting right by twenty-four moves the highest byte down to the bottom, and the cast to char keeps the lowest eight bits. Then we shift by sixteen, and by eight, and the lowest byte goes last, as it is, which is network byte order, written out by hand. We could have used h t o n l, the C function that does it. However, it lives in different headers on Linux and on Windows, and shifting works anywhere. And the body goes after the size, and the frame is ready to send. Append adds the new bytes to the end of what we have. Next is where the waiting happens, and it starts with a check. If the buffer doesn't hold four bytes yet, we can't even read the size, so we return nullopt, the empty optional. Otherwise, we read the size back, the other way round. For each of the four bytes, we shift what we have so far left by eight, and add the byte at the bottom. Each byte goes through a u int eight first, as a char can be signed. So a byte above a hundred and twenty-seven would turn into a negative number, and fill the top of our size with ones. Then we check the size against the limit, k max message size, as a size above it means the stream is broken. So we keep an invalid argument error in status, which is abseil's code for input that's wrong. Its message says how many bytes the size claimed, with std to string, so the log tells us what we got. And we hand back nullopt, leaving the size at the front of the buffer. So each call after this one stops at it too, and returns nothing, as once a size is wrong, there's no telling where the next message starts. Whoever reads from a socket is going to check the status once it's taken the messages out, and close the connection. If the whole message hasn't arrived yet, we return nullopt again, and wait for more. Otherwise, substr copies the message's bytes out, from just after the header. And erase removes the header and the message from the front of the buffer, so the next call starts at the next message. Erasing from the front moves the rest of the buffer along. That's fine with messages this small, and it's something to keep an eye on once we measure performance. In the protocol's CMakeLists, the library gets the framing source. The project's root becomes a public include directory too, next to the build folder's. So the server and the client can include protocol slash framing dot h by its path from the root, like the generated header. The library now links with abseil's status as well, as the frame reader keeps its error in one. It's public too, as whoever reads frames gets the protocol's errors as abseil statuses. And the test executable gets a second file, framing test dot c c. Next to it goes the test file, which starts with our framing header, then string. Then abseil's status, for the error code we expect, and gtest. And the generated messages, as we need real messages to frame. Two helpers, in an unnamed namespace, keep the tests short. Make input builds an Input from an x and a z, so a test can make the message it wants in one call. And x of goes the other way, parsing bytes back into an Input. It expects that to work, as a test that gets bytes it can't parse should fail. Then it returns the x, which is how a test tells which message came out of the reader. The tests are about what TCP can do to our bytes. The first, waits for the whole message, frames an Input with x at one and z at minus one. That's ten bytes for the message, and fourteen with the size in front. We give the reader the first six bytes, the size and a little of the body, as if the rest were still on its way. Next has to return nothing, and not half a message. Then we append the rest, and the message comes out whole, with its x at one. The second test, splits messages that arrive together, is the opposite case. Two frames arrive in one append, one with x at one, and one with x at minus one. That's what happens when the server sends two worlds in a row, and the client reads them in one go. We call next twice, and assert that both came out, before we look at them. They have to come out in the order we sent them, so the first has x at one, and the second minus one. And a third call returns nothing, as there's nothing left over. The last test, rejects messages that are too large, appends four bytes at two fifty-five. That's a size of about four billion, way over our sixty-four kilobyte limit. Next has to return nothing, instead of waiting for four gigabytes to arrive. And the reader's status has to carry the invalid argument code. We check the code, and not the words of the message, so that rewording the message later doesn't break the test. Last, the lint stage in the Dockerfile now runs clang-format and cpplint over the protocol folder too, so its code is held to the style guide like the rest. And in the top-level CMakeLists, the API docs cover the protocol's sources as well.

25:31 Once it's built, let's run the protocol's tests, and all five of them pass, the two from before and the three on framing.

The server's world

25:38 Now, the server, and before any networking, let's write the world it runs, as plain code with no sockets in it. That way, we can test the rules of the game on their own, and quickly, and the networking around them stays a thin layer. Its header and source file sit next to the tick from episode one. After its guard, the header includes chrono, which we used for the tick interval, as the world is going to step by a duration. Then cstdint, for the ids, and map, for the players. Player is a struct, as it's plain data, and it holds where a player stands, as an x and a z. And it holds the input its client last sent, as input x and input z. The input lives in the player because a client sends it when its keys change, and not on each tick. So the world has to remember it from one tick to the next. All four start at zero, so a new player stands still. Then the World class, which is the game as the server sees it. K speed is how fast a player moves, four, like the client had in episode two. It's static constexpr, so it belongs to the class, and not to each world, and the tests can read it too, and its comment gives the unit, in tiles per second. K edge is how far a player's centre can get from the middle, which is five point five. Above it goes the reason: six tiles to the ground's edge, less half a tile, so players stay on it. Then what the server does with the world, starting with join, which adds a player and returns its id. Leave removes one, when its client goes away, and set input stores which way a player wants to move. And step moves everyone along by a given time, such as one tick. Step takes the time in nanoseconds, the type tick interval hands back, so the server can pass its interval as it is. Players hands out all the players, so the networking code can send them. Like the frame reader's status, it's const and returns a const reference, so nobody copies the map, and nobody outside the world can change it. The members are the next id to hand out, starting at one, and the players, in a map from id to player. A map finds a player by its id quickly, which we're going to do with each input that comes in. And it keeps them sorted by id, so the order we send them in stays put. The source file starts with the world's own header, then algorithm, for clamp, and cmath, for hypot and is finite. Join takes the next id, and moves the counter on. Ids start at one, which leaves zero, protobuf's default, free to mean no player at all. New players start in a row across the middle, two tiles apart, at minus three, minus one, one and three. The fifth one starts over at the left, back at minus three. So when a few players join, they don't all land in one spot, and you can tell their cubes apart. Then we add the player to the map, and return its id. Leave erases the player with that id from the map. Set input is where the server takes in what a client sent, so it checks it first. It finds the player, and if there's no such player, we ignore the input. And if either number isn't finite, which means it's infinity or not a number at all, we ignore it too. A NaN would spread through each sum it touches, and the player's position would turn into NaN as well. Then we work out the length of the direction, with hypot, which is the square root of x squared plus z squared. And the comment over it says what it's for: so that moving diagonally isn't faster, and a modified client can't ask for more speed. If the length is over one, we divide both numbers by it, which scales the direction back to a length of one. Holding W and D sends a one on each axis, so the direction is about one point four long. Our client has normalised its direction since episode two, but a modified client might not, so the server does it again. And a client that sends an x of a hundred would move a hundred times as fast. A length under one is left alone, which leaves room for an analogue stick later on. Then we keep the checked input in the player, for step to use. Step turns the time into seconds, as a float, with chrono's duration. Then it goes over the map with a structured binding, which unpacks each entry into its id and its player. The ampersand makes player a reference, so we move the player in the map, and not a copy of it. The new x moves by the input, times the speed, times the seconds, so a player covers four tiles a second, whatever the tick rate. And clamp keeps it between minus and plus k edge, so a player stops at the edge of the ground. Then z works alike, with the z input and the clamp at the edge. Next to the code goes its test file, as we've done since episode one. It includes our world, then chrono, cmath, for hypot again, limits, for a NaN, and gtest. In an unnamed namespace, k tick is the length of a tick at twenty ticks a second, fifty milliseconds. Step takes nanoseconds, and like in episode one, chrono turns the milliseconds into nanoseconds for us, as no precision is lost that way. Run one second steps the world twenty times, by k tick, which is what the server's loop is going to do. So the tests add up small steps, rounding and all, and the numbers we check are the ones the server produces. The first test, moves four tiles a second, joins a player, notes where it starts, and holds right for a second. We expect it to end up four tiles further along, to the right. Expect float equal allows for the tiny rounding errors of adding up a float twenty times, where expect equal could fail on the last digit. The second test, diagonals are no faster, keeps a copy of where the player starts, both its x and its z this time. Then it holds right and towards us at once, for a second, and takes where the player ended up. We expect the distance it covered, measured with hypot, to be four tiles. Without scaling the direction back, it would be about five point seven. The third, keeps players on the ground, joins a player, and sets its input to walk away from us, with minus one on z. Then it runs the world for ten seconds, a second at a time. That would take it forty tiles, so we expect it to stop at minus k edge. Expect equal is fine here, as clamp hands back k edge as it is. And the last one, ignores input that is not a number, joins a player and notes where it starts. Then it sends a quiet NaN as x, from numeric limits. Quiet means this NaN doesn't raise an error when it's used, unlike a signalling one. After a second, the player's x has to be where it started. The server's CMakeLists adds the world to the core library, and its tests to the test executable. Let's come back to set input, as it's the first code where the server checks what a client sent. Anything a client sends can be wrong, by a bug or on purpose, so each message from a client is going to get this kind of check.

32:41 Once it's built, let's run the server's tests, which now cover the world as well. The three on the tick interval from episode one pass, and so do the four on the world.

Networking with Asio

32:51 For the networking, we're going to use Asio, by Christopher Kohlhoff. It's a cross-platform C++ library for networking and other input and output, built around asynchronous operations. It gives us sockets, timers and signals, on top of whatever the operating system does best, which is epoll on Linux, and kqueue on macOS. Asio comes in two variants, and Boost dot Asio is the one that's part of Boost. The standalone one needs nothing more than a C++ compiler, so that's the one we use. It's also what the C++ standard's networking proposals were based on.

33:26 Asio is built around an I O context, which is the event loop all of its operations go through. We start operations, such as accepting a connection or reading from a socket. And we give each of them a function to call once it's done, which is called a completion handler. Then io context run waits on the operating system, and calls each handler as its operation finishes. For our server, the acceptor waits for new clients to connect to our port. Each client gets a session, which reads from its socket and writes to it. A timer wakes us up on each tick, twenty times a second. And a signal set waits for Ctrl C, or for docker stop, to shut the server down. All of these handlers run on one thread, one after another. So two of them can't touch the world at once, and we don't need any locks. A single thread can serve a lot of clients this way, as it spends its time on work, and the waiting happens in the operating system.

34:23 For the networking, we go back to the top-level CMakeLists, and fetch Asio like the others, at version one point thirty-eight point two. Asio comes as headers, which means all of its code is in its headers, with nothing to compile ahead of time. It also doesn't come with a CMake build of its own. So, like we did for ImGui, we describe it to CMake by hand. Add library, with interface, makes a library that builds nothing. It passes its usage requirements on to whatever links it. Here, the requirement is Asio's include folder, which we mark as a system include directory, so that Asio's headers don't set off our warnings either. With that, linking asio is all the server and the client need to do to use it. In the server's CMakeLists, the core library gets three new sources: listen, the session and the server, which we're about to write. The core links with our protocol library, for the messages and the framing, and with Asio. And it gets two more of abseil's libraries, memory and strings, which listen and the server are going to use. It's all public, so the server's executable and its tests get these through the core. Next comes one client's connection, as seen from our side, which we call a session. It reads the client's messages as they arrive, and writes ours to it, in order. After the header guard, the header includes array, for the buffer we read into. Then deque, functional, memory and string, which we're going to see as we go. Then Asio, which has a header for each of its parts. We include the part we use here, asio slash ip slash tcp dot h p p, for the TCP socket. That's called include what you use, which the style guide asks for, as we saw in episode one. Here, it also means the compiler doesn't read all of Asio, which asio dot h p p would bring in. Then our framing header, from the protocol's folder, for the frame reader. The class derives from enable shared from this, a standard class that lets an object make shared pointers that point back at it. A shared pointer counts how many owners an object has, and destroys the object when the last one lets go, which we're going to need for reading and writing. Then two using aliases, which give a type a shorter name. On message is a standard function that takes a message's bytes. Std function can hold anything we can call, such as a lambda, so the server can hand the session its own code. On close is a function that takes nothing and returns nothing. With these two, the session tells whoever made it about messages, and about the connection closing, without depending on the server or the world. That's the separation we want: the session moves bytes, and the game happens elsewhere. The constructor takes the client's socket, and the two functions. Start starts reading from the client, and send queues bytes to write to it. Privately, read, write and close do the work, and the members are the socket, and the two functions. Then a buffer of four kilobytes, as a std array, for Asio to read into. Four kilobytes is plenty for our messages, and the frame reader stitches together whatever arrives in pieces. Then the frame reader we wrote a moment ago, in the protocol library. The outbox is a deque, which is a queue we can add to at the back and take from at the front. That's what we're going to do with the messages waiting to be written. And closed, a flag that makes sure we close the socket and call on close a single time, however the connection ends. The source file starts with its own header, then string, and utility, for std move. Then two parts of Asio: buffer, which wraps our memory for Asio's reads and writes, and write, for async write. The constructor takes the socket and the two functions by value, as the session is going to keep them. Then its initialiser list moves each of them into the session's members. Std move hands an object over to its new owner, without copying it. A socket can't be copied anyway, as two copies would both close the one connection. In the body, we set an option on the socket. Like most of Asio's calls, set option has a version that takes an error code, and fills it in when the call fails. The guide doesn't use exceptions, which is how the other version reports a failure, so we use the error code versions throughout. Here, the error code is called ignored, as the socket works either way, a little slower. Its comment reads: turn off Nagle's algorithm, so small messages go out right away. By default, TCP holds a short write back for a moment, in case more comes, so that it can send them in one packet. That's good for throughput, as a few bigger packets cost less than a lot of small ones. However, our worlds are a few dozen bytes, and we want each one sent as soon as it's ready, without waiting on more. Nagle's algorithm is the name of that holding back, after John Nagle, who came up with it. So we set no delay on the socket, which is TCP's name for switching it off. Start calls read, and read asks Asio for the client's bytes. Async read some starts a read into our buffer, and returns right away, without waiting. Once some bytes have arrived, however many, Asio calls the lambda we gave it, the completion handler. In its square brackets, self equals shared from this makes a variable that lives inside the lambda, a shared pointer to the session. If the read failed, because the client went away or the connection broke, we close, and stop reading. Otherwise, we append the bytes that arrived to the frame reader. Then we take whole messages out of it, in a loop, as one read can bring several, and hand each one to on message. After the loop, we check the reader's status, like we planned when we wrote the frame reader. If it isn't OK, the client sent a size above our limit, so we close the connection, and stop reading. Otherwise, we start the next read, so a read is waiting for the next bytes. Then we go back up, over the whole read, and write down that capturing self, not this, keeps the session alive until the read finishes. Send is what the server calls with each frame, and once we've closed, it does nothing. Otherwise, it adds the bytes to the back of the outbox. Then we look at the size of the outbox, to know whether to start writing. If it's one, the bytes we've just added are all there is, so nothing is being written right now, and we start writing. The check gets a comment of its own: writes mustn't overlap, so if one is under way, it picks these up next. Asio writes a big message in several parts, so two writes at once could mix their parts on the wire, and the client would read garbage. Write starts an async write, of the bytes at the front of the outbox. Unlike read some, async write keeps going until all of the bytes are written. Its lambda captures self too, and if the write failed, we close. Otherwise, we drop those bytes from the front of the outbox, and if more are waiting, we write the next. So the outbox drains a message at a time, in the order the server sent them. A read and a write can both fail when a client goes away, so close checks closed first, and does the rest a single time. It closes the socket, with an error code we ignore, as the socket may be broken already, and it ends up closed either way. And it calls on close, so the server can take the player out of the world. Now, let's go back to self in the read's lambda, and why we didn't capture this. Capturing this would give the lambda a plain pointer to the session, which says nothing about whether the session still exists. When a client leaves, the server drops the session from its map, while a read or a write may still be under way. With a plain pointer, Asio would go on with a session that's been freed. That's a crash if we're lucky, and quietly broken memory if we're not. With self, each read or write under way holds a shared pointer to the session, so it lives until the last of them finishes. Then the last pointer goes, and the session is destroyed with it.

42:28 The server ties it all together, as it accepts clients over TCP, and runs the world at a fixed tick rate. After the header guard, its header includes chrono for the interval, cstdint for the port and the ids, and map and memory for the sessions. Then abseil's status and status or, as starting a server can fail. Then three parts of Asio: the I O context, ip tcp, for the acceptor, and the steady timer. And the session and the world, from the server's folder. Like our window in episode two, the server gets a static function, Create, as a constructor can't hand back an error. It hands back a status or of a unique pointer: the server, or why it couldn't start. Create takes the I O context, the port to listen on, and the interval between ticks. It takes the context by reference, as main owns it, and everything Asio does goes through it. Its doc comment says it starts the server, or returns why it couldn't, such as the port being taken. Create is all the server shows to the outside, as once it's made, it runs on its own, through the handlers it gives Asio. The constructor is private, so Create is the way to make a server. It takes the context and the interval, and does nothing that can fail. Start does the part that can, listening on the port, and returns a status. Then accept, tick and broadcast do the work, once the server is running. The members are the acceptor, which listens for new clients, and a steady timer for the ticks. Steady means it follows a clock that doesn't jump when someone changes the system's time, which matters for a game loop. Then the interval between ticks, the world, and the sessions, in a map from player id to a shared pointer. The map is one of a session's owners, and the handlers Asio holds are the others. To listen, Start has to open the port, and Asio's acceptor can do that in its constructor, given an address and a port. However, that constructor reports a failure with an exception, which we don't use. So opening the port gets a function of its own, which hands back a status instead. It doesn't need anything from the server, so it gets a header of its own, listen dot h. After the guard, it includes cstdint for the port, abseil's status, and Asio's ip tcp, for the acceptor. The function is called Listen, and its first argument is a reference, so it sets up the server's own acceptor, and not a copy. The second is the number of the port, and it hands back a status: OK, or the reason it failed. Then we go back up and document it: opens the acceptor on the port, on all addresses, and starts listening. Any server we write can call it with its own acceptor. In the source file, as in all of our source files, its own header comes first, listen dot h. Next, cstdint, and system error, the standard header for std error code. That's the type Asio fills in when a call fails. Then str cat, from abseil's strings, which joins text and numbers into one string. And Asio's socket base, for the socket's options, as status and ip tcp come with our header. Listen first makes the endpoint, which is an address and a port. Here, it's any IPv4 address, at the port we were given. Any address means clients can reach us from other machines too. That matters in Docker, where connections come in on the container's own network, and not on its localhost. Above it, we say what these are: the steps the acceptor's constructor takes, with the overloads that hand back an error code. Then the error code, which each step fills in if it fails. Open opens the acceptor's socket, for the endpoint's protocol, IPv4. Reuse address lets us listen on the port again right after a restart. Without it, the operating system can hold on to the port for a while after the server stops, as old connections finish closing. Bind ties the socket to our address and port, and that's the step that fails when another program has the port. And listen starts taking connections, with max listen connections, the longest queue of waiting clients the operating system allows. Each step runs if the ones before it worked, and if one of them failed, we return an unavailable error. Unavailable is abseil's code for something that may work if we try again later. Str cat builds its message from the port and the reason in the error code. Otherwise, we return OK status, which is abseil's way of saying it all worked. With that, back to the server, and its source file. Its own header comes first, then iostream, for the log, then memory, string and utility. Then abseil's memory, for wrap unique, while status, status or and Asio's parts come with our header. Then the framing and the generated messages, from the protocol. And our listen dot h, which Start is going to call. Create first makes the server by calling new directly, and hands the pointer to a unique pointer, with abseil's wrap unique, like the window's create in episode two. Then Create starts the server, and if that fails, it returns the status. The unique pointer then deletes the server on the way out. Otherwise, it returns the server, which the status or carries back to main. The constructor gives the acceptor and the timer the I O context, and keeps the interval. None of that can fail, and the acceptor isn't open yet. Start is where the server opens up for clients and starts its ticks. First it listens, with our function, and returns the status if that failed. Then we start accepting, and set the timer to go off one interval from now. And async wait gives the timer its handler, which calls tick. That lambda captures this, the server, and a plain pointer is fine here, as the server lives in main until the loop has stopped. And we return OK, as the server is now listening. Accept starts an async accept, and when a client connects, Asio calls the lambda with an error code and the new client's socket. If that worked, the client joins the world as a new player, and we get its id. Next, an error code called ignored, which we're going to hand to the log line. We log the new player, with the address it came from, so we can see who connected. Remote endpoint takes that error code, like the calls in listen. A client that's gone already gets an address of zeros in the log, and the server carries on. Std endl ends the line and flushes it right away, which matters in Docker, where our output goes through a pipe. Without the flush, the lines would sit in a buffer, and docker logs would show nothing for a while. Then the two functions the session gets, which connect it to the world. On message parses an Input from the bytes, and passes its x and z to the world, for that player. If the bytes don't parse, we ignore them, and the world checks the numbers, as we saw. On close takes the player out of the world, removes its session from the map, and logs that it left. Both lambdas capture this and the player's id, so each of them reports for its own player. And the code that moves the bytes doesn't need to know any ids. Then we make the session with make shared, as it has to live in a shared pointer for shared from this to work. Make shared also puts the object and its count of owners in one allocation. We keep it in the map, under the player's id, and start it. And whatever happened, we accept the next client, so the acceptor keeps listening. Tick steps the world by one interval, and broadcasts it to the clients. Then it sets the timer to one interval after it last expired. That line gets a comment: from the last expiry, not from now, so that the ticks don't drift. Then it waits again, for the next tick to come around. Broadcast builds one World message for everyone, to send after each tick. For each player in the world, add players gives us a new entry in the message, and a pointer to it, like in the protocol's test. Through that pointer, we copy over the id, then the x and the z of the player. The world's player and the message's player are two different types, the server's own and protobuf's, so this is where one turns into the other. Then, for each session, it sets you to that session's player, frames the message, and sends it. We build the list of players once, and change you for each client, instead of building the whole message for each of them. Keep this loop in mind, as we're going to come back to it once we measure the server under load. In main, we include csignal for the signal numbers, and cstdint for the port. Then memory, for the server's unique pointer, string, and utility, for std move. Next comes abseil's numbers, for turning an argument into a number. Then Asio's I O context and its signal set, and our server's header. Main now takes its arguments, argc and argv, which are how many there are, and the list of them. The tick rate and the interval from episode one stay, along with the check that the interval is valid. The port is seven seven seven seven, unless the first argument gives another. Any free port would do, and that one is easy to remember. Simple a to i, from abseil, turns the argument into a number, and writes it into port. It returns false when the text isn't a number, where the C function a to i would quietly return zero. A port also has to fit in sixteen bits, and port zero would ask the system for any free port. So we check that it's from one to sixty-five thousand five hundred and thirty-five. If any check fails, we print that the port should be a number in that range, and return one. Next, we make the I O context, and then we create the server. What Create hands back goes into server or, like window or in episode two. We pass Create the context, the port, and the interval, fifty milliseconds. A port is a sixteen-bit number, so the int goes in as a u int sixteen, and the range check above means it fits. If server or holds an error, we print its status, with the server's name in front, and return one. When it's printed, a status shows its code and its message. So a port that's taken comes out as unavailable, couldn't listen on port seven seven seven seven, address already in use. Otherwise, we move the server out of server or, into a unique pointer that lives until main returns. The log line now says which port we're listening on, along with the tick rate, and the interval from format duration, as before. It ends with std endl too, like the server's other log lines. Next, a signal set, which waits for signals the way a socket waits for bytes. It waits for SIGINT, which is Ctrl C, and SIGTERM, which docker stop sends to ask a program to finish. The comment we add over it says why: as process one in a container, SIGTERM does nothing unless we handle it. In a container, our server is process one, and Linux doesn't apply a signal's default action to process one. So without the signal set, docker stop would wait ten seconds, and then kill the server. The handler is a lambda that captures io by reference, and stops it. And at the end, we log that we've stopped, so docker logs shows it. In the Dockerfile, the server's image exposes port seven seven seven seven. Expose doesn't open anything on its own, and we still publish the port when we run the image. It documents which port the server listens on, for whoever runs the image. One more look at tick, where we set the next expiry from the last one, not from now. The work in tick takes a little time, and Asio calls us a little after the timer fires. So waiting a full interval from now would make each tick a little late, and the ticks would drift over time. This way, twenty ticks take a second, however long each tick took, as long as it took less than an interval.

54:35 Before we put it in Docker, let's build the server here, with dash dash target, so Ninja builds the server and what it needs, and run it. It says it's listening on port seven seven seven seven, at twenty ticks a second, and right after that, that it stopped, and we're back at the prompt. Main made the server, which opened the port, and handed Asio a handler for the next client, and another for the timer. However, Asio runs its handlers inside the I O context's run function, on the thread that calls it, and main doesn't call it. So main got to its end, and returned, and none of the handlers ran.

55:12 So, let's go back to the end of main. The accept, the timer and the signal set each start an operation and return straight away, with a handler for when it completes. Asio calls those handlers from inside run, so without a call to it, the server is all set up, and then main carries on past it. After the signal set, we call run on the I O context. It blocks, and runs the loop, handler after handler, until the signal handler stops it. Once run returns, main carries on to the log line that says we've stopped, and returns zero.

55:48 Let's build it again, which compiles main and links the server, and run it. This time, it keeps running, at twenty ticks a second, until I press Ctrl C. Then the signal set's handler stops the loop, run returns, and the server says it stopped.

56:03 Now let's build the server's image, and on the way, Docker compiles protobuf and the rest on Ubuntu, and runs all fifteen of our tests. Then we run it, with dash p seven seven seven seven colon seven seven seven seven. That publishes the container's port on my machine, so that clients outside Docker can reach the server. The server's log says it's listening on port seven seven seven seven, at twenty ticks a second. When I press Ctrl C, Docker passes the signal on to the server, which stops the loop, and prints that it stopped.

Syncing the cubes

56:36 Now for the client's side, where the client gets a class for talking to the server, Connection, so that main doesn't deal with sockets. It sends our input, and keeps the latest world the server sent us. After the header guard, the header includes array, cstdint, memory and string. Then abseil's status and status or, and Asio's I O context and ip tcp. Then our framing, and the generated messages, from the protocol library. Connecting can fail, say, when the server isn't running or the address is wrong. So, like the server, the connection gets a static Create, which hands back the connection, or why it couldn't connect. It takes a host and a port, and connects, waiting until it's connected. A client that can't reach the server has nothing to show, so it's simpler to wait, and fail right away if it can't connect. Like the server's create, its comment says it connects to the server, or returns why it couldn't. Copying is deleted, like our window's in episode two, as a copy would share the socket. Poll handles whatever has arrived from the server, without waiting for more, and we're going to call it once a frame. Send tells the server which way we want to move, as an Input. Connected turns false once the connection has closed, and world returns the latest world from the server, by const reference, so drawing it doesn't copy it. It's empty until the first one arrives from the server. Privately, there's the constructor, which Create calls, then start, which connects and returns a status. And read keeps the reads going, starting a new one as each finishes. The members are an I O context of our own, the socket, a buffer to read into, and a frame reader. Then the latest world, and a flag for whether we're still connected. The flag starts out true, as by the time Create hands out a connection, it's connected. It turns false once, and stays that way, as we don't reconnect for now. In the source file, the connection's own header comes first, then memory and string. Then abseil's memory and str cat, which we saw in listen. And three parts of Asio: buffer, connect and write, as ip tcp comes with our header. Create works like the server's: it makes the connection with new, and hands it to a unique pointer with wrap unique, as the constructor is private. Then it starts the connection, with the host and the port. If that failed, it returns the status, and otherwise the connection, which is connected by then. The constructor makes the socket on our I O context, in its initialiser list. Start does the connecting, in two steps that can each fail. A resolver turns the host's name into addresses, say, localhost into one twenty-seven dot zero dot zero dot one. Then an error code, for the steps that can fail. Resolve takes the port as text, with std to string, as it also accepts a service's name, such as http. And it fills in the error code when it can't find the host. A name can have several addresses, such as an IPv4 and an IPv6 one. So asio connect tries them in turn, until one of them answers, and fills in the error code if none of them do. If either step failed, we return an unavailable error, like listen does. Str cat builds its message: couldn't connect to, the host and the port, and the reason, such as connection refused. Then we turn on no delay, like on the server, as our inputs are small too, with an error code we ignore. Then we start the first read, and return an OK status. Poll calls poll on our own I O context, the one the socket lives on. Unlike run, it runs the handlers that are ready, and returns right away, without waiting. So the networking happens on the main thread, between frames. We don't need a thread of our own for it, or locks around the world. Send first checks that we're still connected, then it frames the input, and writes it with asio write. That write is synchronous, so it returns once the bytes are handed over. However, a message this small goes directly into the socket's buffer, so it doesn't hold up the frame. We pass an error code, so a failed write comes back in it. If it fails, we mark the connection as closed, and main checks for that on the next frame. Read starts an async read some, like the session's on the server. This time, the lambda captures this, a plain pointer, and that's fine. Its handlers run inside poll, which main calls while the connection exists. If the read failed, we're no longer connected, and we stop reading. When bytes arrive, we append them to the reader, and parse each whole message into our world. A newer world replaces the older one, as all we need is the latest. If a message doesn't parse, the server sent us something we don't understand, so we treat the connection as lost. After the loop, we check the reader's status, like the session does. A message that's too large means we've lost the connection as well. Otherwise, we start the next read, like the session does. The scene header now includes the generated messages, and declares scene, which takes a World from the server. It returns everything to draw for that world: the ground, then a cube for each player, and a marker over ours. It takes the world, and not the connection, so it doesn't depend on where the world came from, and a test could hand it one. And the namespace's closing brace gets its comment, as clang-format adds one once a namespace holds more than a line. In the source file, we include array, for the player colours. K player colours is an array of six colours, known at compile time like the tiles' colours above it. Each one is red, green, blue and alpha, from zero to two fifty-five. The orange from episode two comes first, then blue, green, purple, red and yellow. They're different enough to tell the players apart at a glance. And k marker is white, for the marker over our own player. Scene starts from the ground, from episode two, and then we go over the players in the world. For each one, the position is its x and z from the server, at half a unit up, so that its cube stands on the ground. Then we push back its cube, at that position, with a colour from the list. The colour comes from its id, minus one, so that player one gets the orange. The percent sign wraps the ids around the six colours. We pick it by id, and not by the player's place in the list. So a player has one colour in all the clients, and keeps it when others join or leave. And if the player is ours, which means its id is the world's you, we add a small white cube a unit above it. It's a quarter of the size, and it's a marker, like the ones over a character's head in an MMO. The order of the cubes doesn't matter, as the depth buffer puts the nearer ones in front. The client's CMakeLists adds the connection's source, and the core library links with our protocol library and with Asio, like the server's does. It links with abseil's strings too, for the str cat in the connection. In main, raymath goes, as the client no longer moves anything on its own. We include cstdint, for the port, and string, for the host, along with the connection's header. Main now takes its arguments, like the server's main does. The host is the first argument, or localhost, so you can point a client at another machine. The port is seven seven seven seven, the one the server listens on by default. We connect before opening the window, with connection create, which hands back the connection or an error. So if the server isn't there, we print the status and return one, right away, instead of opening a window on an empty world. Otherwise, we move the connection out of connection or, like the window, and log that we're connected. The window, the GUI, the menu and the renderer stay as they were. The cube's position and its speed go, as the server moves us now. Sent is the input we last told the server, kept from one frame to the next. We keep it so that we send our input when the keys change, and not sixty times a second while they don't. Online is how many players we last saw, for the log. In the loop, we poll the connection first, so whatever has arrived is handled before we draw. If it closed, we print that we lost the connection to the server, and return one. The window closes on the way out, as its unique pointer lets go of it. Then we read the keys, with movement from episode two, and leave the direction as it is, as the server scales it now. If the direction differs from what we sent, we set it on sent, and send it. Then we take the latest world from the connection, to draw it. When the number of players changes, we log which player we are, and how many are online. That's handy for seeing players join and leave in the terminal. And set cubes hands the renderer what scene builds from the server's world.

1:05:34 Let's see it working, with the server running in the terminal at the bottom. When the first client starts, it connects, and the server logs player one joining. Then a second client, which is player two, and a third, player three. Each client shows all three cubes, with the white marker over its own. When player one holds D, its cube moves to the right, in all three windows. Player two moves away from us, and player three towards us, and to the left. Two of them can move at once, and each client sees both. No client moves a cube, not even its own, as each of them sends its keys and draws what the server sends back. You might spot that the movement is a little stepped, which is the twenty ticks a second. Smoothing it out between ticks is a topic of its own, for a later episode.

Docs

1:06:23 Last, we document all of it in the README, for our own good, as next time we won't remember half of these commands. We list the libraries we had CMake fetch for us, and note that the first build takes longer with protobuf. It shows how to run the server, and then a few clients, each in a terminal of its own, on this machine or another. It points at nanommo dot proto as the contract between the two sides. And in the Docker commands, the new one runs the server's image with its port published.

Wrap-up

1:06:53 So, today we made the server the one that decides where everyone is. We wrote a contract between the client and the server, in protobuf, which protoc compiles into both of them, on the Mac, in Docker and in CI. We added framing, so whole messages come out of a TCP stream. The world moves the players and checks what clients send it, with tests. The server runs it with Asio, on a single thread. And several clients now see each other move, each one through the server. We picked TCP to start with, and as Asio does UDP too, we can switch to it once we can measure the difference.

Playing it

1:08:36 Like last time, let me know in the comments what you'd like to see, and which parts you'd like us to go deeper on. And if you want to follow along as the game grows, subscribe to the channel. I hope you find this useful, and thanks for watching.