Prometheus and Grafana for a C++ game server

Intro

0:00 Hello and welcome back to episode four, where we're going to give our server some monitoring, with Prometheus and Grafana. And to have some data to see, we're going to write a bot that plays the game. Let's begin with a quick look at where we're going to be by the end of the episode.

0:17 Three hundred bots are playing on our server, and a Grafana dashboard shows us what it's doing, as it happens. And we're going to see why the bytes we send grow a lot faster than the players.

0:30 On the website of the series you can find more information about the project. The code for the project is open source on GitHub, tagged at each step of each episode so that you can follow along.

Last time

0:41 Now, before we get to today's work, let's quickly go over what we did last time. In the last episode, we started the server, which decides where everyone is in the world. Clients send it the keys they hold, and it moves the players twenty times a second. The client and the server agree on their messages through a contract in protobuf. And each message carries its size, so that whole messages come out of the TCP stream. The server does all of that with Asio, on a single thread. And by the end, three clients see each other move. We also picked TCP to start with, and said we'd come back to UDP once we can measure the difference.

Today

1:19 Today, we're going to look inside the server while it runs. First, the server counts a few things about how it runs, such as how many players are online and how long each tick takes. It serves those numbers as metrics, on a page that other programs can read. Prometheus collects them at an interval of two seconds, and keeps their history. Grafana draws them as a dashboard, so that we can see at a glance how the server is doing. We start all three together with Docker Compose, from a file in the repo. And to give the dashboard something to show, we write a bot that connects like a player and wanders around, and then we start a few hundred of them. All right, let's get started with why we want metrics in the first place.

Why metrics

2:01 So far, all we know about a running server is what it prints. Logs tell us about single events, such as player seven joining. However, they don't tell us how the server is doing overall. We'd like to know whether each tick finishes well within its fifty milliseconds, or gets close to the limit. We'd also like to see how many players are online compared with an hour ago, and how many bytes a second we send to all the clients together. Those are numbers that change over time, and that's what metrics are for. In production, they're how you find out that something's wrong, often before the players tell you. And once we start optimising, they're how we tell whether a change actually helped.

2:42 Prometheus works by pulling the numbers from our server, instead of waiting for the server to push them. The server serves its current numbers over HTTP, at slash metrics, as plain text. At a regular interval, two seconds apart for us, Prometheus fetches that page, which is called a scrape. It stores each number with the time it read it, and a number with its history like that is called a time series. Grafana then asks Prometheus questions in PromQL, which is Prometheus's query language, such as the bytes sent per second over the last five minutes, and draws the answers. Pulling keeps the server simple, as it doesn't need to know where Prometheus is, or whether it's up.

3:22 Prometheus is an open-source monitoring and alerting toolkit, which started at SoundCloud in 2012. It's now a project of the Cloud Native Computing Foundation, which also looks after Kubernetes. Its architecture has more pieces than we need today, such as the Alertmanager, which sends alerts when a number crosses a line, and service discovery, which finds what to scrape. For now, we use the Prometheus server in the middle, which scrapes and stores the time series, and Grafana, at the bottom right, to look at them.

3:50 Prometheus has a few types of metric, and we're going to use three of them. A gauge is a number that goes up and down, such as the players online. A counter is a total that goes up, such as the bytes we've sent since the server started, and it's reset to zero when the server restarts. On its own, the total isn't very interesting, as it keeps going up for as long as the server runs. However, Prometheus can work out how fast it grows, which gives us bytes per second. And a counter doesn't miss anything that happens between two scrapes, while a gauge of bytes per second would. A histogram is for durations, such as how long each tick takes. With twenty ticks a second, there are forty of them between two scrapes. So the histogram counts them into buckets: how many took up to a tenth of a millisecond, how many up to a quarter, and so on. From the buckets, Prometheus estimates percentiles, such as the fiftieth percentile, or p50, which is the typical tick. The ninety-ninth, or p99, is the time that ninety-nine ticks in a hundred stay under. An average would hide a few slow ticks among a lot of fast ones, while the p99 shows them.

Metrics in the server

5:01 To serve metrics from C++, we're going to use prometheus-cpp, by Jupp Müller. It's the client library for Prometheus that most C++ projects use. Its README shows the whole idea in one example, which we're going to follow in our server. An exposer is a small HTTP server that serves the metrics, and a registry holds them. Each metric is made with a builder, from a name and a help text, and then it's registered with the registry. And whenever Prometheus scrapes the exposer, the exposer reads the registry and serves what's in it.

5:35 We start in the top-level CMakeLists, under Asio, where prometheus-cpp comes in with a few options of its own that are on by default, and we don't need them. So we turn them off before we fetch it, and its CMakeLists picks up our values. Enable push would build a library that pushes metrics to a gateway. That's meant for short jobs, which finish before Prometheus could scrape them. Our server runs for hours, and Prometheus is going to pull from it, so push is off. Enable compression would gzip the metrics page, and it needs zlib for that. Our page is a few kilobytes, so we keep zlib out of the build. And enable testing would build the library's own tests, and add them to our test runs, so that's off too. Then we declare it with FetchContent, like GoogleTest in episode one. The URL points at version one point three point oh, at a tarball from its releases, called prometheus-cpp with submodules. The exposer is built on CivetWeb, a small web server, which the repo keeps as a git submodule. A plain download of the source would leave it out, while this tarball bundles it. SYSTEM marks its headers as system headers, like protobuf's, so that warnings from its code don't fill up our build. And make available fetches it, and adds its targets to ours. In the server's CMakeLists, the core library gets metrics dot c c. It links with prometheus-cpp's core target, which has the metric types and the registry. The server's executable also links with the pull target, which has the exposer and its web server. We keep the two apart on purpose, so that our tests can link with the core library and nothing more. That way, they check our metrics without a web server, and without opening a port. And the test executable gets a third file, metrics test. Now a new header, metrics dot h, for a class of our own that holds the server's metrics. The rest of the server is going to talk to this class in its own words: players, ticks and bytes. So prometheus-cpp stays behind this one file, where the rest of the server can't see it. If we add a label or change library later on, the rest of the server doesn't have to change. It opens with its header guard, then chrono, which we've used since episode one, for how long a tick takes. cstddef, for size t, the type we use for the number of bytes we send. And memory, for shared pointers, as the registry is going to be held in one. Then prometheus-cpp's headers, one per type of metric, in quotes with their folder, like Abseil's. Counter dot h is for the bytes and the messages, and gauge dot h for the players online. Histogram dot h is for the tick, and registry dot h for the registry, which holds the metrics and hands them to whoever reads them. In our namespace goes the class Metrics, and its constructor is going to set up all four metrics. Set players takes how many players are online, and the server calls it when that changes. Observe tick records how long a tick took, as a chrono duration in nanoseconds, so the unit travels with the number, like the tick interval in episode one. Add bytes sent adds to the bytes we've sent to clients. And add message received counts one message from a client. Registry returns the registry, for the exposer in main to serve. It's an accessor, in lower case and const, like the frame reader's status, and it returns a reference to the shared pointer, so there's no copy. Then the private members, and the registry comes first, in a shared pointer. That's because the exposer is going to keep a pointer to it too, so they share it. Then a reference to each metric: a gauge for the players, a histogram for the tick, and counters for the bytes and the messages. The registry owns the metrics, and hands us a reference to each one as we add it. A reference member has to be set in the constructor, and it can't point anywhere else later, which suits us here. And the order matters, as members are set up in the order they're declared. So the registry exists before we add anything to it. The source file includes metrics dot h, from the root of the project like all our includes, and goes into our namespace. The Metrics constructor does all its work in its initialiser list, as that's where references have to be set. The registry comes from make shared, which creates it together with the shared pointer that owns it. Then the players, where build gauge starts a builder for a gauge. We give it a name, nanommo players online, in snake case, like Prometheus's own metrics. The nanommo prefix keeps our metrics together, and apart from anybody else's, once Prometheus scrapes more than our server. The help text, players connected to the server, is what Prometheus shows next to it. Register adds a family of gauges to our registry, and Add makes the gauge in that family, for a set of labels. Labels are names and values that split a metric into several series, say one per region of the world, later on. We don't have any yet, so the braces are empty. Next, the tick, as a histogram, and its name ends in seconds. Prometheus's convention is to use base units, so seconds are what we're going to observe. Its help text is time spent in each tick, and this time, Add also takes the bucket boundaries. Each boundary is two or two and a half times the one before. So the buckets are narrow at the fast end, and wide at the slow end, where a small difference matters less. Its comment gives the range: from a tenth of a millisecond up to the whole fifty-millisecond tick. That's the range we care about, as faster than that, the server is hardly doing anything. Slower, the tick has taken longer than it's allowed, and it lands in an extra bucket for everything above. Then the two counters, where nanommo bytes sent total counts the bytes queued for clients. That's what we hand over to be sent, as the writes happen a little later. And nanommo messages received total counts what the clients send us. By convention, a counter's name ends in total, which tells whoever reads it that it's a counter. Now the functions, starting with set players, which sets the gauge. prometheus-cpp keeps all its values as doubles, like Prometheus does, so the count goes through a static cast. Observe tick turns the nanoseconds into a duration of seconds, held in a double, and observes that number. Add bytes sent increments the bytes counter by the size. And add message received increments its counter, and with no argument, increment adds one. CPPLINT dot cfg, the file of cpplint settings we started in episode one, gets a second filter, which turns off the check called whitespace, indent namespace. Then we go back up and add a comment on why we turn it off. cpplint two point oh point two reads braces inside a constructor's initialiser list, such as dot Add with empty braces, as the start of a block indented inside the namespace. clang-format keeps namespaces unindented already, so we leave that check to it. The test file includes our header first, then chrono, for the tick, and string, for the text we scrape. Then gtest, and prometheus-cpp's text serialiser, which writes metrics out in the text format Prometheus reads. Two helpers go in an unnamed namespace, starting with scrape, which collects what's in the registry and serialises it. So it returns what Prometheus would get if it scraped the server right now. Has tells us whether the scraped text has a line we're looking for. It looks for the line with a newline after it, so that a count of one doesn't match a count of ten as well. Then the test, which is called puts each tick in its bucket. We make the metrics, observe a tick of three hundred microseconds, and scrape the text. Three hundred microseconds falls between the quarter and the half millisecond boundaries, so it shows whether the tick lands in the right bucket. We expect the count to be one, and the bucket for up to a quarter of a millisecond to have zero. And we expect the bucket for up to half a millisecond to have one. Buckets are cumulative, so each one counts all the ticks up to its boundary. Those lines have quotes inside them, so they're raw string literals, like our shaders in episode two. This test pins the part we could get wrong: the conversion to seconds, and the buckets. If observe tick passed on microseconds as if they were seconds, our tick would land in the bucket for slow ticks. And the percentiles on the dashboard would be off by a factor of a million. The gauge and the counters hand their numbers directly to prometheus-cpp, which has tests of its own, so they don't get a test here.

14:19 Now that we have a Metrics class, let's have the server report to it. In the server's header, we include metrics dot h, next to session and world. Create takes the metrics as a fourth argument, by reference. And the private constructor takes them too, as Create hands them over when it makes the server. The server keeps that reference as a member, as it doesn't own the metrics. Main is going to make them, and hand them to the exposer as well, so they live in main, next to the server. We could have made them a global, which any file can reach. Passing them in keeps it clear who uses them, and a test can give a server metrics of its own. In the source file, Create takes the metrics as well. It makes the server from the I O context, the interval and the metrics, through wrap unique, like before. The constructor takes the metrics, and sets the reference in its initialiser list, after the interval. Then in accept, when a player joins, we set the players online to the size of the world's players. We could also go up by one here, and down by one when a player leaves. Taking the real count each time means the gauge can't drift, even if a join or a leave got counted twice somewhere. In on message, we count the message first, before we parse it. That way, the count takes in messages that fail to parse as well, and a client that sends us garbage still shows up on the graph. When a player leaves, once it's gone from the world and the sessions, we set the players again. Then in tick, start is the time now, from the steady clock, like our timer's in episode three. The system clock can jump when the computer syncs its time, and that would give us a negative or a huge tick now and then. We step the world and broadcast, as before, and then we observe how long that took: the steady clock's now, minus start. Subtracting two points in time gives us a duration, which goes into nanoseconds without a cast. And in broadcast, we frame each client's message into a string of bytes first. We add its size to the bytes sent, and then move it into the session's send, so it isn't copied. That's why we need the local, as once the bytes are moved, we can't ask them for their size. In main, we include prometheus-cpp's exposer, and our metrics header, which sort in among the others. Then the metrics port, nine four six four, so the metrics get a port of their own, next to the game's seven seven seven seven. The game's port carries our framed messages, while the metrics are served over HTTP, so sharing a port would mean one port speaking two protocols. And in production, the game's port is open to the internet, while the metrics port stays inside, where Prometheus can reach it. We make the metrics, and then the exposer, which is prometheus-cpp's small HTTP server. It listens on zero dot zero dot zero dot zero, which means any of the machine's addresses, so that Docker can forward the port to it, like the game's port. Register collectable hands it our registry, which the exposer serves from two threads of its own, while our server runs on its single thread. That's safe, as prometheus-cpp's metrics are made to be updated on one thread and read on another. The metrics are declared before the server, so they're there for as long as the server uses them. Create gets the metrics as its fourth argument, after the interval. And we log where they're served, next to the port we listen on. In the Dockerfile, the server's image exposes the metrics port as well, next to the game's. Looking at tick once more, the time we measure covers stepping the world, and building and queueing each client's message. The writes to the sockets happen afterwards, when Asio gets to them. So this is the cost of the tick on its own, not all the work the server does. And we're going to keep that in mind when we read the numbers.

18:09 Once it's built, let's run the server's tests, and the new metrics test passes. Along with the three tick tests and the four world tests from before, that makes eight. Then I've started the server in the background, and curl fetches its metrics page, the way Prometheus would. Grep keeps the lines about players, where hash HELP is our help text and hash TYPE says it's a gauge. The value is zero, as nobody has connected to the server yet. The second grep shows the tick's histogram, where each bucket's line has an l e label, which stands for less than or equal. With nobody to move, each tick takes less than a tenth of a millisecond, so all the buckets count all of them. The sum is their total time, and the count is how many ticks there have been.

A bot that plays

18:55 For these numbers to move, we need players, and a lot more of them than I can play by hand. So, let's write a bot that connects like a client, holds a direction for a while, like someone at the keys, and then picks another. It goes into a new folder, bot, next to the server and the client. We start with the decisions a bot makes: which way to go, and for how long. They go in a file of their own, as plain functions with no network, so that we can test them on their own. Wander dot h starts with its header guard, then chrono, for the time a bot holds a direction before it turns. Random, which is the standard library's header for random numbers. And nanommo dot p b dot h, our generated messages, for the input a bot sends. Random direction returns an Input, with x and z each at minus one, zero or one, which is what W, A, S and D send. It takes an mt19937, by reference, which is the Mersenne Twister, a well-known random number generator from the standard library. We pass it in, instead of making one inside the function. We could also use the C function rand, but that keeps a single hidden generator for the whole program. With ours passed in, each bot has its own, and a test can seed one so that its numbers repeat from run to run. Random hold returns how long a bot holds that direction, in milliseconds, from its generator too. The source file includes wander dot h, from the bot's folder, and opens our namespace. Inside random direction, a uniform int distribution goes from minus one to one. A distribution turns the generator's raw numbers into the values we ask for, here minus one, zero or one, each as likely as the others. Taking the remainder by three would favour some of them, ever so slightly, so we leave that to the standard library. The input is the message the client sends, from last episode. We draw x, and then z, and set them on the input, cast to floats, as that's what the message carries. Both can be zero, which is a bot standing still for a moment, like a player who lets go of the keys. Random hold makes another uniform distribution, from five hundred to three thousand, where both ends can come up. It draws a number of milliseconds from it, and returns them as a chrono duration. So a bot holds a direction for half a second to three seconds. That's about how long you'd hold a key before changing your mind, and it's long enough for a bot to cover some ground. The tests include wander dot h first, then chrono and random, as the tests make their own generator. Set, which is the standard library's sorted collection that keeps each value once. And utility, for pair, which holds two values together, here an x and a z. Then gtest, for the test macros, like in the server's tests. The first test is called holds directions like the keys would. We make a generator, seeded with one, so that the test draws its numbers in the order it did last time. Seen is a set of pairs of x and z. We draw a thousand directions, and insert each one's x and z into the set. A thousand draws are plenty to come across all the combinations. Then we expect the set to have nine of them, three values of x times three of z. The nine has a u after it, which makes it unsigned, like the set's size. That way, the comparison doesn't trip our warnings about mixing signed and unsigned numbers. Then, for each x and z in the set, we expect them to be minus one, zero or one. The square brackets are a structured binding, like in the world's step in episode three. Comparing floats with equals is safe here, as whole numbers like these are stored without rounding. So the test pins what a player at the keys could do: all nine directions, standing still among them, and nothing else. The second test is called holds for half a second to three seconds. Again, with a generator seeded with one, it draws a thousand holds. For each hold, expect G E checks that it's greater than or equal to five hundred milliseconds. And expect L E checks that it's less than or equal to three thousand. So the test would catch a distribution that goes past either of them, say if we'd typed the bounds the wrong way round. Chrono compares the durations for us, with their units, so there's no converting to do. The bot's CMakeLists follows the server's, with a core library, an executable and the tests. The core library, nanommo bot core, gets the bot's sources: bot dot c c, which we're going to write next, and wander dot c c. Its public include folder is the project's root, like the server's, so our includes start from there, as in bot slash wander dot h. It links with our protocol, for the messages and the framing, and with Asio, for the network. It doesn't link with the client, so the bot has no raylib and no window. As such, it builds and runs anywhere the server does, on a machine with no screen or in a container. The executable, nanommo bot, is main dot c c, linked with the core library. Then the tests, nanommo bot tests, linked with gtest main as well. And gtest discover tests registers them with CTest, like the others. And in the top-level CMakeLists, we add the bot's folder, after the server, and the API docs cover its sources too.

24:22 Now the bot, which works a lot like the client's connection from last episode. Its header starts with its guard, then array, for a buffer of a fixed size to read into. cstdint, for the thirty-two-bit type of the seed each bot starts from. Random, for the generator that decides where each bot goes next. And string, which holds the bytes of each message we send. Then Asio, a header for each part we use, as in the server. The I O context runs the bots, TCP has the socket, and the steady timer is for the holds. The class Bot is a pretend player, with no window and no keyboard. Its constructor takes the I O context, which all the bots are going to share, and a seed for its random numbers. It doesn't touch the network, so it has no error to report, and a plain constructor does the job, without a Create. Copying is deleted, with equals delete, like our window's in episode two. A bot owns a connection, and its handlers point back at it, so a copy of it couldn't work. Start dials one of the server's addresses, and then starts wandering. The addresses come resolved, as main looks them up once for all the bots. Connected tells the swarm whether this bot is in the game. Privately, read and wander do the work, and we're going to write them in the source file. Then the members, starting with the socket and a steady timer. Random is the generator, seeded for each bot, and the buffer is an array of sixty-four kilobytes. The server sends each bot the whole world, twenty times a second, so the buffer is sixteen times the client's, and each read takes in more at once. Outgoing holds the bytes we're sending, until the write has finished with them. And connected starts out false, until the connection is made. It's a plain bool, as the bots and their handlers all run on one thread, so there's no other thread to read it while it changes. The source file includes bot dot h, and iostream, to log when a bot can't connect. Then the parts of Asio this file uses, starting with buffer, which wraps our bytes for a read or a write. Connect has async connect, while the I O context and TCP come with our header. And write has async write, which sends our bytes in the background. Then wander dot h, for our two functions that pick a direction and a hold. And framing, to put each message's size in front of it, like the client does. The constructor sets up the socket and the timer on the I O context, and seeds the generator. Start is where the bot connects, to the addresses that main resolved. It calls async connect, which tries them in turn, until one of them answers. The client connected and waited, as it has a single connection to make. A bot can't wait, as hundreds of them share one thread. So it connects asynchronously, and Asio calls the lambda once it's done. The lambda's second argument is the address it connected to. We don't need it, so it's an unnamed const auto reference. If there's an error, we log why, and that bot gives up. Otherwise, we make an error code called ignored, and set no delay, like in the client. We mark the bot as connected, and start reading and wandering. Then read, which the bot needs, although it doesn't look at the bytes. Its comment says that we ignore the world, but still read it so the server's writes don't back up. Inside, async read some reads whatever has arrived into the buffer. When it completes, we check for an error first, as a failed read means the connection is closed. So the bot isn't connected any more, and we cancel the timer. Cancelling makes the wait that's under way finish with an error, and that stops the wandering. Then we return, without reading again, and that bot is done. Otherwise, we drop the bytes, and start the next read, like the session does. Wander frames a random direction into outgoing, and sends it with async write. Outgoing is a member, and not a local, as async write needs the bytes to stay alive until they're written, and that's after wander has returned. The next write comes half a second later at least, and a write of a few bytes is done long before that, so they don't overlap. If the write fails, the bot isn't connected any more. Then the timer waits for a random hold, which comes from random hold in wander dot h. Once it fires, and as long as there was no error and we're still connected, we wander again, in a new direction. In the bot's main file, the headers come first, starting with chrono, for the ramp. csignal, for the signals that stop us, like the server's main, then cstdint and iostream. Memory is for unique pointers, which are going to own the bots. String, utility for std move, and vector, for the list of bots that the swarm keeps. Then Abseil's numbers header, to read the numbers we're given, like the server's port. Then Asio's parts: the I O context, TCP for the resolver, the signal set, and the steady timer. And bot dot h, our own header, for the class Bot we just wrote. Then an unnamed namespace, for a class that's private to this file: the Swarm. It starts the bots one at a time, spread evenly over a ramp. That way, the server sees players arrive over time, the way real ones would, and not three hundred connections in one instant. Its constructor takes the I O context, the server's addresses, how many bots, and the ramp, in milliseconds. It keeps the context, moves the addresses in, and keeps the count. Gap is the time between two bots, the ramp divided by the count. Chrono can divide a duration by a number, and the result is still in milliseconds. The question mark guards the division, as asking for zero bots would otherwise divide by zero. Then the two timers, one to add bots and one to report. And the constructor adds the first bot, and starts reporting. Add starts by comparing the bots we have with the count we want, and once they're all there, it returns without adding more. Otherwise, it makes a bot with make unique, which allocates the bot and gives us a unique pointer to it. The seed is the bot's number, so each bot wanders its own way, and on the next run, each one picks its directions in that order again. We start the bot, and push it onto the vector, with std move. The vector holds pointers, and not the bots, because a vector moves its elements when it grows. And each bot's handlers hold on to its address, through this. With pointers, the vector moves the pointers around, and the bots stay where they are. Then the timer waits for the gap, and adds the next bot, as long as the wait didn't end with an error. Report goes through the bots, and counts the ones that are connected. The question mark adds one for a connected bot, and zero for the others. If that count is different from the one we reported last, it logs how many of the bots are connected, and remembers the count. That way, the log shows the bots arriving, and it doesn't print the count again each second while nothing changes. Then the report timer waits for a second, and when it fires, it reports again, so this goes on until we stop the swarm. The members are the I O context, the server's addresses, the count, the gap, the two timers, the bots, and the count we last reported. In main, the arguments are the host, the port, how many bots, and the ramp in seconds. The host defaults to localhost, and the port to seven seven seven seven. Both stay as text, as that's what the resolver takes. The count starts at a hundred bots, and the ramp seconds at ten. When they're given, simple A to I, from Abseil, reads each one into its int, like the server's port. If either of them isn't a number, we print one message that says both should be numbers, and return one. Then chrono turns the ramp's seconds into a duration, which the swarm takes in milliseconds, without losing anything. We make the I O context, and a resolver, which turns the host into addresses, once for all the bots. Then an error code, called error, which resolve takes as its last argument, and fills in when it can't find the host. What it returns are the server's addresses, which the bots are going to share. If there's an error, say a typo in the host's name, we log that we couldn't find it, with Asio's reason, and return one. We log what we're about to do: how many bots, against which host and port, and over how many seconds. Then we make the swarm, which adds its first bot right away. The signal set waits for SIGINT and SIGTERM, which is Ctrl C and docker stop, and stops the I O context, like the server does. And run runs hundreds of bots on a single thread, as each bot is no more than a few handlers, waiting for Asio to call them. Once the I O context has stopped, we log that, and return zero. In the Dockerfile, a new stage, bot, starts from a plain Ubuntu image, like the server's stage. It copies nanommo bot from the test stage, so there's no bot image unless the tests pass. It runs as the ubuntu user, not as root, like the server's image does. And its entrypoint is nanommo bot, so the arguments we give the container go directly to it. Further down, the lint stage finds the bot's folder too, so clang-format and cpplint check its code like the rest. Let's come back to read, which the bot can't skip. If it stopped reading, its socket's buffers would fill up before long. And the server's outbox for that bot would keep growing, as the server queues a world for it on each tick. A real client that stops reading, say because its machine froze, would do that too. So it's a case the server is going to have to handle at some point.

34:17 Let's try a few of them first, before we go big with hundreds. The server is running at the bottom, and a client is connected, so that we can watch. Next to the server, nanommo bot starts forty bots, spread over eight seconds. They join one by one, and the server logs each of them. In the client, their cubes appear and wander around the board, each one holding a direction for a moment and then picking another. Some of them bunch up along the edges, as the server keeps everyone on the board. The bots send their keys like any client, and the server moves them.

Docker Compose

34:53 We now have three programs to run together, the server, Prometheus and Grafana, and the bots when we want load. We could start each of them with a docker run of its own, with its ports and its files, and remember all of that each time. Instead, Docker Compose describes all of them in one file, compose dot yaml, which we keep in the repo. Compose puts them on a network of their own, where each container reaches the others by its service name. So Prometheus scrapes the server at server colon nine four six four, and Grafana asks Prometheus at prometheus colon nine zero nine zero. The bots are in the file too, and they stay off until we ask for them. And docker compose up starts the lot with one command, on any machine with Docker: mine, yours, or a server later on.

35:39 Docker's documentation describes Compose as a tool for defining and running multi-container applications. The services, networks and volumes of the stack all go into a single YAML file. It comes with Docker Desktop, and on Linux, it's a plugin that Docker's packages install along with the Docker engine.

35:56 compose dot yaml goes at the top of the repo, where docker compose looks for it by default. Name sets the project's name, nanommo, and Compose names the containers and the network after it, such as nanommo server one. Without it, the project would be named after the folder we run it from, and that changes from one checkout to another. Then the services, starting with the server, where build says Compose builds its image from this repo. The dot is the context, which is the files the build can see. Target picks the server stage of our Dockerfile, which is the image CI builds too. Ports publishes two of the container's ports on my machine, host port first and container port second. That's seven seven seven seven for the game, and nine four six four for the metrics. They're in quotes, as Docker's documentation recommends, because YAML can read two numbers with a colon between them as a number in base sixty. Next, Prometheus, which comes as an image, prom slash prometheus, pinned to version three point fifteen point oh. Volumes mounts our config file into the container, at the path where Prometheus looks for it. The colon r o at the end mounts it for reading, so the container can't change our file. And port nine zero nine zero is its web interface. Last, the bots, which are built from this repo too, from the bot target. Command passes nanommo bot its arguments, and the host is server, which is the service's name on Compose's network. Then the port, three hundred bots, and a ramp of a hundred and twenty seconds. Depends on makes Compose start the server before the bots. And profiles puts them in a profile of their own, called load. A service in a profile stays out of docker compose up, until we ask for it by name, or turn its profile on. So the stack comes up quiet, and we add the load when we want it. Prometheus's config goes into a new folder, monitoring, which is going to hold Grafana's files too. Global holds the defaults for all the jobs, and the scrape interval is two seconds. Prometheus's default is a minute, and fifteen or thirty seconds is common in production. Prometheus stores each number it reads, and a large setup can have thousands of targets. We go for two, so that the graphs react quickly while we watch. Under scrape configs, we add a job called nanommo server, for our server's metrics. Prometheus puts the job's name on each series it scrapes, as a label, so we can tell our numbers apart from any other job's. Static configs is a fixed list of targets, and ours is localhost colon nine four six four, where we read the metrics with curl. Prometheus can also discover its targets, say from Kubernetes, but a fixed list is enough for us. It asks for slash metrics by default, which is where our exposer serves them. And the dot docker ignore file gets compose dot yaml and the monitoring folder, so that editing them doesn't rebuild the server, protobuf and all.

38:55 I've built the images already, as that's the slow part. Docker compose up dash d starts the stack, where dash d is short for detached, so it runs in the background. Compose creates the network, and starts the server and Prometheus. The bots stay out, as they're in the load profile. Docker compose up dash d bots starts them, by name. Docker compose logs bots shows what the bots have printed so far. They're joining the server, two or three of them a second. And the server's metrics page, on the port we published, counts them already.

Prometheus

39:32 Prometheus has a web interface, on port nine zero nine zero. Its targets page lists what it scrapes, and our server is down. The error says that connecting to localhost, port nine four six four, was refused. And yet, our curl read the metrics from that address a moment ago. The curl ran on my machine, where nine four six four is the port we published. Prometheus runs in a container of its own, and in a container, localhost is that container. So Prometheus asks its own container for our metrics, and nothing there listens on that port.

40:07 So, back in Prometheus's config, the target has to point at the server's container. Compose puts our services on a network of their own, and on it, each one is reachable by its service name. The bots reach the server that way, as we gave them server as the host. So the target becomes server colon nine four six four, which Compose's network resolves to the server's container. The ports we publish are for programs on my machine, like curl, while containers on the network reach each other's ports directly.

40:37 Prometheus reads its config when it starts, so we restart it, with docker compose restart prometheus. The config file is mounted from the repo, so the container sees our change without a rebuild.

40:50 Back on the targets page, our server is up, and Prometheus scraped it a moment ago. On the query page, nanommo players online draws the bots as they join, a couple of them a second. Now for a counter, where the PromQL function rate turns nanommo bytes sent total into bytes per second, over the fifteen seconds in the square brackets. And the line curves upwards, as it grows faster than the players do, and we're going to see why, once there's a dashboard.

Grafana

41:18 Prometheus's own graphs are fine for a quick query, but for a dashboard that we keep coming back to, we use Grafana. It's open source, and it draws from Prometheus and a lot of other sources. Its provisioning system sets up data sources and dashboards from files, and those files can go into version control. So, instead of clicking a dashboard together in the browser, where it lives in Grafana's database, ours is going to be a file in the repo. And changes to it go through git like any code. A data source file tells Grafana where Prometheus is, and a dashboards file tells it which folder to load dashboards from.

41:55 In compose dot yaml, Grafana gets a service of its own, between Prometheus and the bots. It comes from the grafana slash grafana image, pinned to thirteen point two point three, like Prometheus. Its settings come from environment variables, as Grafana can take any of its settings from a variable named after it, starting with G F. So we don't need a config file of our own. Anonymous enabled lets visitors in, without an account or a login. The value is in quotes, as an environment variable is text, and YAML would otherwise read true as a boolean. The anonymous org role says what those visitors can do. Viewer means they can look at dashboards, and they can't change anything. On our own machine, that saves logging in each time. On a server that's open to the internet, we'd turn it off. The default home dashboard path opens our dashboard when we go to Grafana, from the file we're about to write. Then come two volumes, and the first mounts our provisioning folder, where Grafana looks for its set-up files: the data sources and the dashboard providers. The second mounts our dashboards folder, which holds the dashboards as JSON files. Both are mounted for reading, like Prometheus's config file, and port three thousand is Grafana's web interface. The data source file goes under provisioning, data sources, and Grafana reads the files in that folder when it starts. API version one is the version of the file's format. Then a list of data sources, with one in it, called Prometheus. Its uid is a fixed id, prometheus, and our dashboard's panels refer to their data source by that id. Without it, Grafana would make up a random one, and the dashboard wouldn't find its data. Type says it's Prometheus, and the URL is http colon slash slash prometheus colon nine zero nine zero, the service's name on Compose's network. Is default makes it the data source that new panels use. And under JSON data, where the settings for this type of data source go, the time interval is two seconds, our scrape interval. That way, Grafana doesn't ask for finer detail than Prometheus has. Under provisioning, dashboards, a provider called nanommo loads dashboards from files. Its path is the folder where Compose mounts our dashboards, inside the container. Grafana looks at that folder for changes, so when we edit the file, the dashboard follows. And the dashboard is a long JSON file, so I'm pasting it in. Grafana can export it for us, once we've laid a dashboard out in the browser. Its uid is nanommo server, which goes into its URL, and its title is nanoMMO server. It shows the last five minutes, and refreshes on a five-second timer. Then come four panels, and players online is a stat panel, a single big number, from nanommo players online. Tick duration is a time series, drawn from two queries on the tick's histogram. Sent to clients is the rate of the bytes sent, in bytes per second. And messages from clients is the rate of the messages received. Now the tick's two queries, where histogram quantile point five estimates the fiftieth percentile. It works from the rate of each bucket over fifteen seconds, so it describes the last fifteen seconds of ticks, and not all of them since the server started. Fifteen seconds holds seven or so scrapes, and rate needs at least two. Sum by l e adds the buckets up across all the series that share a boundary. With one server and no labels, there's one series per bucket, so it doesn't change the result yet. Once there are labels, or several servers, it's what keeps the percentile working. And the second query passes point nine nine instead, which gives us the p99.

Three hundred bots

45:42 With the stack up and the bots on their way, Grafana opens on our dashboard. Three hundred bots are joining, over the two-minute ramp we set in compose dot yaml. Players online is our gauge, and it climbs as they join. Messages from clients grows in proportion to the players, so twice the bots means twice the messages. Each bot sends a new direction after a random wait of up to three seconds, a little under two seconds on average. The bytes we send are different, as that line curves upwards, faster and faster as the players come in. And the tick duration climbs along with the bytes we send. With a few dozen players, the p99 is around two point two milliseconds. With two hundred and ninety-five players, it's around twelve milliseconds, which is already about twenty-four percent of our fifty-millisecond budget.

46:31 The bytes grow that fast because on each tick, the server sends each player the whole world, with all the players in it. With a hundred players, that's a hundred messages, of a hundred players each, so ten thousand player updates on each tick. With three hundred, it's three hundred times three hundred, which is ninety thousand. So three times the players means nine times the work, which is called quadratic growth, and it's in our broadcast. Games deal with it by sending each player what's near it, which is called interest management. However, before changing anything, I want to push the server until it actually struggles, and measure where that happens. So we're going to come back to this, with numbers.

Docs

47:11 Last, let's make sure the README documents all of this, so that we don't have to work it out again later. It lists prometheus-cpp among the libraries CMake fetches, and the docker run command publishes the metrics port too. Then a new section covers monitoring and load, with the metrics endpoint, docker compose up, where to open Prometheus and Grafana, and how to run nanommo bot. It also mentions that on a Mac, a few hundred bots need a higher limit on open files, as each connection is a file, and that's ulimit dash n.

Wrap-up

47:44 So, today our server started measuring how it runs, with a gauge for the players, a histogram for the tick, and counters for what it sends and receives. A test pins the histogram's buckets, so that a mistake in the units shows up right away. Prometheus scrapes them, and Grafana draws them on a dashboard that lives in the repo. One compose file starts all of it, on any machine with Docker. We wrote a bot that plays like someone at the keys, and started three hundred of them. And the dashboard showed us that what we send grows with the square of the players. With load that we can generate, and numbers to read, we're also in a position to compare TCP and UDP properly, later on.

Playing it

49:24 Let me know in the comments what you'd like to see, and what you'd like to measure. And if you want to follow along as we build it, subscribe. I hope you find this useful, and thanks for watching.