C++ game server with CMake, tests, Docker and CI

Intro

0:00 This is an MMO we're building from scratch, with a client and a server in C++. The server decides where everyone is, and each player sees the others move.

0:09 And around it, the engineering: Docker, tests and CI, and monitoring, here with three hundred bots playing at once. This first episode is where all of it starts.

0:20 Hello and welcome. In this series, we're going to build an MMORPG from scratch, both the client and the server. We're going to focus somewhat on the gameplay, and try to add some cool little features along the way. Hopefully, some of them are going to be your suggestions. The plan is for this to be an extensive series, which is going to cover a lot of topics.

0:43 I've always had an interest in game development. I've used Unity and Unreal. And over the last ten years, I've had multiple attempts at building my own game engine. As for my background, I studied computer science for my undergrad. Then I worked as a software engineer, for big tech and other large companies, but not as a game developer. And now, I'm doing a PhD in artificial intelligence, specifically in reinforcement learning for enhancing reasoning in generative models. Since I haven't worked as a game developer, what I want to focus on primarily is what I have direct experience with. That's the software engineering and DevOps side of creating something like this, applied in a domain I'm passionate about. The purpose is to help you learn what comes into play in a production-like setup, especially if you're at the start of your career. That said, maybe over time, we're going to find ways to apply my research in this project as well.

1:37 For example, AI society experiments. Say, Generative Agents: Interactive Simulacra of Human Behavior, which is a paper from Stanford. They put twenty-five agents, each driven by a language model, in a small town called Smallville. The agents wake up, go to work, and talk to each other. And when one of them was given the idea of throwing a Valentine's Day party, the others spread the invitations, asked each other out, and turned up together at the right time. Another example is reinforcement learning, for animations and movement.

2:09 As for MMORPGs themselves, my experience is as a player. I played Cabal Online, Aion and Metin2 back in the day. I haven't actually played World of Warcraft. And I was quite a fan of purely browser-based text games in general. What fascinates me the most is how advanced the computer science behind these games is. It's often underappreciated by the people playing them. That's often the case, and it makes sense why it's like that, as it's all meant to stay behind the scenes. So again, that's the reason I want to create content like this. I want to contribute to showing how much effort humanity has put into the technology that allows all of this to exist. And I want to show how that effort can continue in the age of AI.

2:52 Before we go any further, a quick note on how this series is made. Sometimes, the voice you're hearing is generated with AI. It's designed to sound like me, and the wording follows how I speak, so none of it is left to chance. It's all highly moderated. And many hours go into designing the curriculum, informed by my own experience from industry and from my personal projects. Using AI simply lets me make better quality content, which is my goal. It means I don't necessarily have to aim for a perfect recording. Instead, I can say more of what I realise is meaningful to show you. So it's human effort to a large extent, and I'm leveraging tools that let me scale it. Each part of the project is added deliberately, in order to share knowledge and help you understand complex systems. And a lot of it goes well beyond what you'd naturally do in a personal project.

3:48 Visually, it's going to be a polygonal, low-poly style of game. As such, we're going to lean on procedural generation, with code building the world and a lot of what's in it. So we're not going to spend much time making assets by hand in tools such as Blender. The main reason is that I'm not that great at it. But also, the series is designed with the engineering and the computer science as its main focus. So I'm leaving the rest up to the imagination of those of you who follow along, and want to put some of this into practice yourselves.

The series

4:21 When it comes to game development, there are some really high quality creators out there. One of them is The Cherno, who's been making a game engine called Hazel. He goes really in depth into C++, and into how a game engine works behind the scenes. These days, he's the CTO at a robotics company, where he's building a game engine for training AI robots. So if the engine side is what you're after, I'd recommend his channel.

4:46 And there's Elegon, a new MMORPG inspired by World of Warcraft Classic. It's being built with Godot as the engine, and SpacetimeDB behind it. SpacetimeDB is a database that also runs the server's logic, so it's quite a different take from the one we're going to go with here.

5:03 Besides those, there are a lot of series out there on gameplay development, on modelling, or on building a game engine. And there are others on topics such as monitoring, as a general principle. What I want to explore here is the skillset you need in order to work on games deployed at scale. And more generally, on distributed systems and production software, which several teams have to be able to work on, and which has to last for years. So, besides building the game, its client and its server, we're also going to use Docker, so that it builds and runs on almost any system. And we're going to have automated builds and tests whenever something changes. We're going to benchmark it with simulated players, collect metrics with Prometheus, and have monitoring with Grafana. We're going to talk about our database choices, with Redis for the fast-changing state and MySQL for persistence, along with its migrations. And we're going to go into the nitty-gritty and do some performance optimisations, all the way from the C++ and the graphics to the server communication. Along the way, we're also going to focus on data structures and algorithms, and on building systems that account for scale as much as possible. We won't have the feedback loop that comes with deploying this to tens of thousands of real players. But we're going to do our best to mock it in our performance testing, if nothing else. And it's all going to be open source, so you can play with it and run it yourself.

6:27 For now, the working title of the project is nanoMMO. It takes after nanochat, a repository by Andrej Karpathy. nanochat trains a small ChatGPT-like model and lets you talk to it, end to end, with code that's kept minimal and easy to hack on. And the whole pipeline fits in a fairly small set of files, which you can actually read through. That's what I'm going for here as well. A codebase that covers an MMO from one end to the other, which you can read, run yourself, and play with.

6:56 So your takeaway won't necessarily be how to build the best game engine or the best database schema. Instead, it's going to be how to be an end-to-end engineer who understands the whole pipeline. One purpose of this series is to build timeless skills, which hold up as we move to an AI-first world. Debugging, in-depth understanding and holistic engineering are going to stay a core need. And I believe society is going to keep rewarding them. However, the expectations are going to be that much higher. It's now much easier for someone without experience to fake it till they make it. But in the long term, that's never going to pay off, compared to someone with real understanding. Someone like that works in synergy with assistive tools, and not as a bystander. So when a problem occurs, you're able to debug it and think about what's actually happening behind the scenes.

Architecture

7:47 So, let's look at the architecture. Players are going to run a native client, written in C++, with raylib for the graphics. It connects to the game server, which is also written in C++. I've written before about keeping business logic out of the UI layer, and it applies here as well, with a heavy bias towards a thin client. As such, the server is going to be authoritative. It runs the simulation at a fixed tick rate and takes care of all the brainy stuff, while the client takes care of input and rendering. In a game, this matters even more, as we can't trust a client that's running on someone else's machine. For the messages in between, my proposal is Protocol Buffers. Behind the server, Redis is going to hold the hot state, such as sessions and positions. And MySQL stays the source of truth. I'm not going to go deep on these two today. We're going to bring each of them in once the game actually needs it, so that you see the problem it solves first. Say, once we have a few hundred simulated players, and saving each move straight to the database puts too much load on it. Picking SQL is mainly my preference. NoSQL databases, such as MongoDB, would also work for a game like ours, so we're going to dig into that when we get there, and we might change our mind by then. And to see what's going on inside, the server exposes metrics, which Prometheus scrapes and Grafana turns into dashboards. Before we write any code, I want to go a bit deeper on protobuf, as we're going to lean on it from the start.

Why Protocol Buffers

9:18 Protocol Buffers, or protobuf for short, is a format for serialising structured data, which came out of Google. That means turning a message into bytes that we can send over the network. And the side that receives them turns those bytes back into the message. In the docs, it's described as being like JSON, but smaller and faster. We describe each message once, in a dot proto file. Each field has a type, a name and a number. On the wire, it's the number that identifies the field, not its name. The proto compiler then generates code from that file, for C++ and a lot of other languages. So the client and the server both compile code generated from one schema, and that schema becomes the contract between them. Neither side writes its own encoding or decoding by hand, so the two can't drift apart. And adding a new field doesn't break older code, as it skips the fields it doesn't know. And unlike JSON, protobuf is a binary format. Say we have a message with a single number field, and we set it to a hundred and fifty. Protobuf encodes it in three bytes, one that says which field it is, and two for the value. In JSON, it's text, with the field's name in quotes and braces around it, so nine bytes. And the receiver has to parse that text back into a number. With a lot of messages going to a lot of players, many times a second, that adds up. We could also write our own raw binary protocol, which is small and fast. But we'd be writing the encoding and the decoding by hand, on both sides. And we'd have to keep them in step ourselves. And adding a field would break older clients. So protobuf gives us most of the size and speed of a raw protocol, along with a contract that both sides compile.

Running it anywhere

11:06 Last, I want to touch on where all of this is going to run. The server is going to ship as a Docker image. And it's going to expose its metrics in Prometheus's format. That's on purpose, as it lets us deploy to AWS, Google Cloud, Microsoft Azure, or any other cloud provider. Each of them offers a monitoring stack of its own, such as CloudWatch on AWS, or Azure Monitor. However, they all integrate with Prometheus, and with the other standard tools you'd end up using, such as OpenTelemetry. So my proposal is to plan for monitoring from the start. Think about what you actually want to monitor. Say, the time each tick takes, the number of players online, or how long the database takes to answer. And then lean on an open protocol for it. That way, the cloud providers have usually done the work for you. They're compatible with it, and they provide at least their own collectors for structured metrics. They also take deployments as Docker images, which follow an open standard as well, from the Open Container Initiative. And it goes beyond the cloud. It could be a research cluster at a university, or a custom environment inside a studio, small or large. A small studio is likely to use these standard tools as they are. A big one might have in-house tools instead. However, it would still follow similar practices, adjusted to its own needs.

Today's plan

12:25 As for this first episode, we're going to set up the project in a healthy way, such that it's not in our way as we go along. We start with a tiny server and a single function worth testing. CMake is going to build it, and GoogleTest is going to test it. Then we move the whole build into an Ubuntu image, so that it doesn't matter whose machine it runs on. The server gets a small image of its own, which we can run anywhere Docker runs. And we have GitHub Actions run it automatically for us whenever we push. And that pipeline is also going to publish documentation, generated from the code itself. All right, let's get started.

A CMake project

13:02 We start from an empty repository. For building it, we're going to use CMake, which is the standard build system when it comes to C++. CMake doesn't actually compile anything itself. Instead, it generates the build files for a tool such as Make or Ninja. That's what lets one project build on a Mac, on Linux or on Windows. The top-level CMakeLists asks for a recent CMake and names the project, with C++ as its language, at version zero point one. Then we set the rules for everything inside it. We want C++20, and we mark it as required. That way, a compiler that can't do C++20 stops the build with an error, instead of quietly falling back to an older standard. And we turn off any compiler-specific extensions, so that the code stays portable. We export a compile commands file, which editors and tools read in order to understand how each file is built. We put all the executables in a single bin folder, so we always know where to find it. And we switch on the warnings from day one: all, extra and pedantic. We do it now, because they get more expensive to fix the longer we wait. Then add subdirectory pulls in the server folder. The server's own CMakeLists adds a single executable for now, built from main.cpp. Main includes iostream, the standard library's header for printing to the terminal. And main prints a line and exits. Then a bit of housekeeping. A clang-format file, so that all the C++ in this project follows Google's style, and nobody has to format anything by hand. An MIT licence, as it's all open source, so you're free to take this code and build on it. And last, we tell git to ignore the build folder.

14:45 Let's configure the project. Dash S tells CMake where the source is, and dash B where the build folder goes, so that everything it generates stays out of the source tree. Dash G picks the generator, and we're going to use Ninja. Ninja is a small build tool, which came out of Google's work on Chrome. It reads the build files CMake generated, and runs the compiler on whatever has changed, as fast as it can. Make does a lot more than that, so it spends longer working out what to do. And Ninja uses all your CPU cores by default, while Make needs to be asked to with dash j. So our builds are quicker, especially when we've changed a single file. We pick the generator once. After that, CMake knows to run Ninja for us. Then we build it, and run it. It's not much of a game server yet, but it compiles, and we know where everything ends up.

A first function

15:38 Now for some actual logic. An MMO server doesn't react to each packet the moment it arrives. Instead, it advances the world in fixed steps, called ticks, a set number of times per second. So our first function turns a tick rate into the time between two ticks. And for time, C++ has chrono, which is the standard library's date and time library. Cppreference documents all of the standard library, and it's well worth keeping a tab open on it. Chrono gives us clocks, which have a starting point and a rate at which they tick. We're going to need those later on, to time the server's loop. Chrono uses the word tick for its clocks, which has nothing to do with our server's ticks. Today, we need durations. A duration is a number of ticks of some unit. Say, forty-two seconds is forty-two ticks of one second. So the unit is part of the duration's type, and we're going to lean on that in a moment. And since C++14, chrono comes with literals. They let us write a unit straight after a number, such as m s for milliseconds.

16:44 The header starts with pragma once, which stops it from being included twice into one source file. The function is going to return a time, so next, we include chrono, for its durations. And everything we write is going to live in the MMO namespace. The comment above the function is in Doxygen's format, which we're going to use for the docs later on. It documents the contract. It takes the number of ticks per second and returns the tick interval, accurate to the nanosecond. And it throws if the rate isn't positive, as a rate of zero or less doesn't make sense. The function itself returns nanoseconds, which is one of chrono's durations. As we saw in the docs, a duration carries its unit as part of its type. So we can't mix up milliseconds and nanoseconds by accident, and the compiler takes care of converting between them. The alternative is a plain integer, where the unit lives in nothing more than a variable name or a comment. Say we pass milliseconds to a function that expects seconds. It still compiles, and the server runs a thousand times too slow. With chrono, that mistake either gets converted for us, or doesn't compile at all. The source file includes our header first. Then stdexcept, the standard exceptions header, for the invalid argument we throw. Using the chrono literals we saw in the docs, a thousand milliseconds is written as a thousand followed by m s. The function throws an invalid argument for anything but a positive rate. Otherwise, it returns a thousand milliseconds divided by the tick rate. In CMake, the function goes into a small core library, which the server links against. Soon, the tests will too. Keeping the logic out of main from the very start keeps the door open for testing it in isolation. Main includes chrono, to convert the interval, and our header, to get it. It asks for the interval at twenty ticks per second. Ticks per second is a constexpr, which means its value is known when the code compiles. Then it converts the interval to milliseconds with a fractional part, just for printing it. So the implementation itself is quite simple, a thousand milliseconds divided by the tick rate. Keep this line in mind, as we're going to come back to it.

19:00 Let's rebuild and run it. Twenty ticks per second gives us fifty milliseconds per tick, as expected. This number is important, as it's our budget. Everything the server does in a tick has to fit inside those fifty milliseconds. And a good part of this series is going to be about keeping it that way, as more and more players join.

Unit tests

19:21 To test it, we're going to use GoogleTest, which is the most widely used test framework for C++. It's also where GoogleMock lives, which lets a test swap the parts of the code it doesn't care about for fakes. Today, we're going to use two of its features. It finds and runs our tests for us, so we don't have to list them anywhere ourselves. And it gives us a rich set of assertions, such as checking that two values are equal, or that a call throws.

19:48 We don't install GoogleTest by hand. Instead, we let CMake fetch it for us. FetchContent downloads a pinned release when the project is configured, and builds it along with our own code. That way, nobody ends up building against a different version. Enable testing then switches on CTest, which is CMake's own test runner. In the server's CMakeLists, we add a test executable, which links the core library together with gtest main. That gives us a main function which runs all the tests, so we don't have to write one ourselves. Gtest discover tests then registers each test case with CTest, one by one. The test file includes our header first. Then gtest dot h, GoogleTest's header, which gives us its test macro and its assertions. And stdexcept, for the invalid argument we expect it to throw. Using the chrono literals again lets the expected values read like the durations they are. Each test is written with GoogleTest's test macro. The macro takes the name of a suite and the name of the case. Inside, expect equal compares what we got with what we expected. Twenty ticks per second should be fifty milliseconds, which is the case we've already seen working. Sixty ticks per second is a common rate for fast-paced games. It should be sixteen point six six six milliseconds, accurate to the nanosecond, as the header comment says. We write it in nanoseconds, with digit separators to keep it readable. And expect throw checks that invalid rates throw an invalid argument, so zero and minus twenty.

21:24 The first build also compiles GoogleTest itself, so it takes a little longer. Then we run the tests, and one of them fails. At sixty ticks per second, we get sixteen milliseconds, when it should be sixteen and two thirds. When we checked the output by eye earlier, it looked fine, as twenty divides a thousand evenly, but sixty doesn't. As such, each tick would come out two thirds of a millisecond short. And the server would run about four percent too fast, without an error anywhere.

A bug, caught

21:54 So, let's see why it failed. When we divide a chrono duration by a number, the result keeps its unit. As such, a thousand milliseconds divided by sixty is sixteen whole milliseconds. The remainder is gone before the conversion to nanoseconds ever happens. To fix it, we start from one second, converted to nanoseconds, and then we divide it by the tick rate. The integer division still rounds, but now it rounds to the nanosecond, which matches the header comment.

22:23 Let's rebuild, and this time run the whole suite through CTest, which is what we're going to rely on from now on. All three tests pass. And more importantly, that test now guards this line for as long as the project lives. If anyone simplifies it back to milliseconds a few episodes down the line, CI is going to catch it before it reaches the players.

Building in Docker

22:43 Right now, this builds on my Mac, with Apple's compiler. However, the server is going to run on Linux, so that's where it should be built and tested as well. And Docker lets us do that, on almost any machine. A Dockerfile is the recipe for an image, and ours has three named stages. The toolchain stage starts from Ubuntu 24.04 and installs build essential, which brings in GCC. It also installs CMake, Ninja, and the certificates that FetchContent needs in order to download GoogleTest over HTTPS. It then deletes the package lists, as all they'd do is make the image bigger. The build stage copies the source in, configures a release build, and compiles it. The test stage runs CTest, printing the output of any test that fails. So if a single test fails, the whole Docker build fails. The dockerignore keeps the git history and any local builds out of the image. Each stage starts from the one before it, and we can stop at any of them with dash dash target.

23:47 Let's build the test target. Plain progress prints all the log lines, so we can see what's happening inside the container as it goes. The packages install, GCC compiles the project, and at the end, our three tests pass again, this time on Linux.

Running it in Docker

24:02 So far, Docker builds and tests the server, but it doesn't give us anything we can run. And that's what we want in the end, as an image is what we're going to deploy later on. The build image, however, is far too heavy for that. It carries the compiler, CMake, GoogleTest and all the files the build produced. So we add one more stage, called server. It starts again from a clean Ubuntu 24.04, which is the release we built on, so the C++ standard library our binary needs is already in there. Copy dash dash from takes files out of an earlier stage, and we take nothing but the server binary. User switches from root to the ubuntu user, which comes with the image. That way, if anyone ever manages to break into the server, they don't get root inside the container. And the entrypoint is the command that runs when the container starts, so running the image means running the server. The binary comes from the test stage, not the build stage. As such, Docker has to run the tests before it can build this image. So if a single test fails, we don't get a server image at all.

25:07 Let's build it. Dash dash target picks the new server stage, and dash t gives the image a name, mmo-server, so that we can refer to it later on. The toolchain comes straight from Docker's cache, so what gets compiled and tested again is our own code. Then the server stage copies the binary over. Docker run starts a container from the image. Dash dash rm removes the container once the server exits, so that stopped containers don't pile up. The server prints the line we saw earlier on my Mac, this time from inside a Linux container.

API docs

25:42 Next, documentation. Doxygen is the standard tool for generating documentation from C++ code. It reads the comments in our source, along with the classes, functions and variables they sit on, and turns them into a website. It can also produce PDF, through LaTeX, as well as XML. However, HTML is what we're going to use. Inside a comment, Doxygen has commands such as param, return and brief, which describe a function's parameters and what it returns. They start with a backslash or an at sign. The header comments we wrote earlier use the at sign, so they're already in its format. It also understands C++ quite well, so it picks up namespaces, classes and templates, and links them to each other.

26:26 In CMake, find package looks for Doxygen, and we set up the docs if it's installed, so that a machine without it can still build and test. However, Doxygen's default look is quite dated. So we first fetch a more modern theme called doxygen awesome, pinned to a release, like we did with GoogleTest. Then come the settings, each of them a variable which CMake passes on to Doxygen. A name and a short description, an output folder inside the build, and the README as the front page. Extract all documents everything, even code that doesn't have comments yet. The tree view moves the navigation into a sidebar, which is what the theme expects. And the colour style stays on light, since the theme brings its own dark mode. Last, doxygen add docs creates a docs target, built from the README and the server's sources. In the Dockerfile, Doxygen joins the toolchain, and a docs build stage builds that target. A final stage then starts from scratch, which means a completely empty image. It copies in nothing but the HTML. The README explains how to build, test and run the server, and how to generate the docs, with or without Docker. And the output folder is called site, and it's ignored by both git and Docker.

27:41 With dash dash output, Docker copies the files of the final stage out of the build, and doesn't produce an image. What we get is a folder of plain HTML, which we can host anywhere we like.

27:54 The README has become the front page, so the first thing anyone reads is how to build the project. We get search and navigation in the sidebar, as well as a dark mode which follows your system. Our one function has its own entry. The description, the parameter, the return value and the exception all come straight from the header comments. It's a small start, but any function we add from now on is going to end up here. And since the docs are generated from the code whenever we build, they stay up to date with it.

CI with GitHub Actions

28:24 Finally, continuous integration, and first, one more Docker stage. It starts from a bare Ubuntu image with nothing but clang-format, finds all the C++ sources and headers in the server, and checks them. Dash dash dry run reports what it would change, without changing it. And dash dash W error turns any difference into an error, so the stage fails if even a single file isn't formatted the way the style file says. GitHub Actions picks up workflows from the workflows folder under dot github. Ours runs on pushes to main, on pull requests, and on demand, which is what workflow dispatch means. The build job runs on GitHub's latest Ubuntu machine. It checks out the code and runs the commands a developer would run by hand. That's the formatting check, the Docker test build, the server image, and then the docs build. And at the end, it uploads the site for the next job. That second job deploys the docs to GitHub Pages, with no more permissions than it needs. Those are to write to Pages, and to prove its identity while doing so. The README gets a line for the new check as well. Note the conditions on the docs job. It needs the build to pass first, and it runs on main and nowhere else, so the published docs always match the latest episode. And CI runs nothing more than the Docker build you'd run on your machine. So when CI fails, you can reproduce it locally, and you don't have to guess what went wrong.

Wrap-up

29:54 So, from now on, each change is going to be built, tested and documented automatically, on any machine, without us having to think about it. And that's our project set up in a healthy way, ready for the fun part!

30:06 This is going to be an evolving project, so I'd love to hear from you. Let me know in the comments which direction you'd like the game to take, and which parts of the engineering you'd like us to go deeper on. And if you want to follow along, subscribe. Thanks for watching.