Our dashboard shows how the worlds are doing, as long as someone is looking at it. This time Prometheus alerts us on what players would notice: a world that falls behind, where everyone slows to a crawl, and a world that crashes, which throws its players back to the login screen. We watch the alerts go pending and fire on Prometheus's alerts page, find out why up == 0 lets go of a crashed world once Consul drops it, and write a rule that holds on. Then Alertmanager emails each alert to Mailpit, a mail server in Docker, which we read from the terminal with curl and jq, and promtool tests both rules in CI. This is episode 30 of a series about building an MMORPG from scratch in C++, with the focus on the engineering around the game.
The code
Each step on screen is a tag in the game's repository. The code at the end of the episode is ep030/end.
Transcript
Intro
0:00 Hello and welcome back to episode thirty, where we're going to have Prometheus tell us when one of our worlds falls behind or goes down, and have each alert sent to us by email. Let's begin with a quick look at where we're going to be by the end of the episode.
0:16 On Prometheus's alerts page, both of our new rules are firing. Ember has three hundred bots on a twentieth of a core, so it runs a few ticks a second, out of its twenty, and Frost has crashed, while Alertmanager emails us about each of them.
0:32 On the website of the series you can find more information about the project. The code for the project is open source on GitHub, tagged at each step of each episode so that you can follow along.
Last time
0:43 Now, before we get to today's work, let's quickly go over what we did last time. The world server got a function that returns the port it listens on, so that a test can start it on port zero. The server's tests start a world server inside the test, and play it with players over TLS, on the test's thread. UBSan found that our fixture destroyed its I O context before the server, which the plain build ran past. Two players see each other until one of them leaves, and an account plays once at a time. And the coverage report now has the server's game loop running in our tests.
Today
1:18 In episode nineteen, our VPS fell behind past a few hundred players, and we saw it because we were running a benchmark against it at the time. We can't watch a dashboard all day, so today, we're going to have our monitoring tell us when players would notice that something's wrong. Prometheus can check rules that we write in PromQL, and we're going to have one for a world that falls behind, where everyone slows to a crawl, and one for a world that's down, which throws its players out. We're going to see each of them in the game first. An alert goes from pending to firing once its rule has held for long enough, and Prometheus's alerts page shows where each one stands. Then Alertmanager takes the alerts that fire, and sends them as emails, to a mail server that runs in Docker, next to the game. That way, we can check the whole path of an email without a mail account, and without sending real mail. And last, the rules get tests, which CI runs, as a rule is code too, and a change to one should fail the build before it reaches the VPS. All right, let's get started, with what a world that falls behind looks like in the game.
Falling behind
2:22 Alice is in Ember, with a hundred and fifty bots playing around her, while the world runs its twenty ticks a second, and everyone moves the way they should. Now we take nearly all of Ember's CPU away: its threads move onto one core, at the lowest priority, and a busy loop starts on that core. Within seconds, everyone slows to a crawl: the bots' legs keep running, while each tick moves them a little, a few times a second. And alice starts and stops late, as the server applies her keys on its next tick. So the first alert we write is going to be for a world that falls behind like this.
Alerting rules
2:57 Prometheus's documentation on alerting rules starts with an example, where an alerting rule has a name and an expression in PromQL. Each series that the expression returns becomes an alert, with that series' labels. The for clause has Prometheus wait that long before the alert counts as firing, and until then, the alert is pending, so that a short spike doesn't fire it. Annotations hold longer text for whoever reads the alert, such as a description, or a link to what to do about it. Further down, the templating example has an alert for an instance that's down, when up is zero for five minutes, where up is a series that Prometheus records for each target it scrapes.
3:36 Our rules go in a new file in the monitoring folder, alerts dot y m l, next to Prometheus's config. Rules come in groups, and Prometheus checks the rules of a group one after the other, on a schedule. Ours is called nanommo, with the game's two rules in it. The first alert is world falling behind, for a world that can't keep up. A world ticks twenty times a second, and once its ticks take longer than their fifty milliseconds for a while, it manages fewer of them. Then its players see everyone slow to a crawl, the way alice did a moment ago. Our tick histogram counts each tick, so the rate of that count over thirty seconds is how many ticks a world runs each second, like the ticks per second panel on our dashboard. Sum by world keeps the world's name as the alert's label, and drops the address of its container, which can change when the world starts again. And when that rate is below nineteen, the world is falling behind. We pick nineteen, as the rate of a world that keeps up wobbles a little either side of twenty, and at twenty, the alert would keep going pending. For one minute is how long the rate has to stay below nineteen before the alert fires. The minute is longer than the thirty seconds the rate is taken over. A world that has just started has fewer than thirty seconds of ticks behind it, so its rate reads low until it has them. The summary is a template, with the world from the alert's labels, and the value, printed with one decimal, out of twenty. The second alert is world down, like the documentation's example. Up at zero for a minute is a world that Prometheus has failed to scrape for that long, and the summary names it. In prometheus dot y m l, the evaluation interval is how often Prometheus checks the rules. The default is a minute, and we make it five seconds, so that an alert goes pending within seconds. Rule files lists alerts dot y m l, which Prometheus finds next to its config. And in compose dot y m l, Prometheus's container gets the rules file next to its config, mounted like the config.
Pending and firing
5:43 I've built the images already, so docker compose up dash d dash dash wait starts the login server, both worlds and Prometheus, and returns once they're up. Then we start the bots, three hundred of them by default, which arrive over the thirty seconds that ramp says, all of them in Ember. Our machine keeps up with a lot more players than that, so like in the clip, we take CPU away from Ember. In Compose, docker update changes the limits of a running container, and dash dash cpus zero point zero five leaves Ember's container a twentieth of a core.
6:16 Prometheus's alerts page lists our group, from alerts dot y m l, with its two rules. World falling behind is pending, as Ember has been below nineteen ticks a second for less than a minute so far. Opening the rule shows its expression, the minute it has to hold for, and the template of its summary. Below them is the alert, with Ember in its labels, the time since it went pending, and the ticks a second that Prometheus read, a lot fewer than twenty.
6:46 A minute later, world falling behind is firing, and it stays firing for as long as Ember is behind. World down is inactive, as Prometheus scrapes both of our worlds without a problem.
A world that crashes
6:58 For the second alert, alice goes for a walk in Ember, with sixty bots around her. Then Ember crashes, which we do with kill dash nine, like in episode thirteen. Alice's client drops back to the login screen, as it lost the connection to the server, and so does each player who was in Ember.
7:18 In Compose, docker compose kill server sends Ember a SIGKILL too, so let's look at Prometheus's alerts page while Ember is down.
7:27 A minute or so after the crash, world down is firing. Prometheus can't reach Ember, so up reads zero for it, and the alert has Ember in its labels, with a value of zero.
7:39 Less than a minute later, world down has no alert any more, and both of our rules are inactive, while Ember is still down. The targets page shows why: the nanommo server job has one target left, Frost. Since episode twenty-one, Consul drops a world whose check has been critical for a minute, and Prometheus discovers our worlds through Consul, so it stopped scraping Ember. Up has no sample for Ember any more, so there's no zero for the rule to find, and the alert resolved. When we took the documentation's rule, we missed that our worlds leave the list of targets, while the example suits machines that we list by hand.
8:15 A world that stops on purpose, as in a deploy, deregisters on its way out, so for it, up at zero wouldn't fire at all. So the rule has to see a world that has gone from the targets, as well as one whose scrapes fail. The new expression is long, so it goes on two lines, in a folded block, which YAML joins with a space. The rule starts from each world that Prometheus has had as a target in the last day, with max over time of up, over a day, which is one for a world that was up at some point. Max by world folds a world's series into one, as a world that starts again can come back at another address. Then unless takes away the worlds that are up now, and on world has it match the two sides by the world's name alone. What's left is each world that's down, whether its scrapes fail or it has left Consul. A world that we retire for good keeps alerting for a day, until its last sample is older than that. The summary says what the rule checks now: the world is down, or gone from Prometheus's targets.
9:17 With the new rule, world down fires a minute after Ember crashes, and two minutes in, with Ember gone from Consul and from Prometheus's targets, it's still firing. Its alert has Ember as its label, and a value of one, which is Ember's up at its highest over the last day.
Alertmanager
9:34 Prometheus decides when an alert fires, and sends it on to Alertmanager, a separate program from the Prometheus project, which decides who gets told about it, and how. It groups alerts of one kind into a single notification, so that ten worlds going down at once make one email, instead of ten. Inhibition holds back an alert while another one fires, such as the worlds' alerts while the machine they all run on is down. And a silence mutes alerts for a while, from Alertmanager's web page, say while we work on a world.
10:05 Alertmanager sends its email through a mail server, and on our machine, that's going to be Mailpit, an email testing tool by Ralph Slooten. It acts as an SMTP server, SMTP being the protocol that mail is sent with, and it keeps each email it gets instead of sending it on, with a web page to read them. It has a REST API too, for programs and tests to read the emails with.
10:31 Alertmanager's config is a new file in the monitoring folder, alertmanager dot y m l. Global holds what our receivers share, starting with the smarthost, which is the mail server that Alertmanager hands its email to: Mailpit's SMTP port, one zero two five, on Compose's network. The email comes from alerts at nanommo dot example, as dot example is a top-level domain set aside for examples. Mailpit runs without TLS on our machine, so SMTP require TLS is false. The route decides what happens to each alert, and ours sends them all to a receiver called email. Group by alert name puts the alerts with one name into one group, so that both of our worlds going down make one email. Group wait is how long a new group waits for more alerts before its first email, which we make ten seconds, down from the default of thirty, so that the email comes soon after the alert fires. Group interval is how long a group waits before it emails about a change, such as an alert that resolves. We make it a minute, down from the default of five, so that the resolved email comes soon after the world is back. Repeat interval sends an alert that's still firing again after four hours, so that it comes back to the top of the inbox. The email receiver sends to on call at nanommo dot example. Send resolved sends an email when an alert resolves, so that we know when the world is fine again. And text is the plain text part of the email, a template with the summary of each alert on a line of its own. In Prometheus's config, alerting points Prometheus at Alertmanager, by its service name, on port nine zero nine three. In compose dot y m l, Alertmanager gets its official image, pinned like the others, with our config mounted where it looks for it, and its web page on port nine zero nine three. Mailpit gets its image, with its web page and its API on port eight zero two five. Its SMTP port isn't published, as Alertmanager reaches it on Compose's network. On the VPS, both listen on localhost, like Prometheus and Grafana, and come back after a crash or a reboot. Mail that Mailpit gets there stays on the VPS, where we'd read it through the SSH tunnel. To get these alerts on your phone, the smarthost would be a real mail server, such as your email provider's, with a user name and a password, which Alertmanager can read from a file.
12:56 Let's start the worlds again, with Alertmanager and Mailpit this time, and crash Ember like before. A little over a minute later, Mailpit has an email, so curl asks its API for the messages, and jq prints the subject of each one: firing, with one alert, world down, for Ember. The latest message's text is the line that our template wrote: Ember is down, or gone from Prometheus's targets. Then docker compose start server brings Ember back, and within a minute or so, a second email says that the alert has resolved.
Testing the rules
13:31 So that a mistake like up at zero fails CI, the rules get tests, which promtool runs. Promtool is Prometheus's command-line tool, and its test command checks rules against series that the test writes out, without a Prometheus. The test file, alerts underscore test dot y m l, lists the rule file it tests, and checks the rules each five seconds, like our config. The first test group is Frost falling behind, with a sample each two seconds. Its input series are the tick counters of two worlds, and their values are written as a start, plus a step, times a count. Ember's counter goes up by forty each two seconds, for sixty steps, which is twenty ticks a second, and Frost's by thirty, which is fifteen. Two minutes in, the alerts that fire have to be this list, and no more: world falling behind, for Frost, with its summary. So Ember, at twenty, mustn't fire, which shows that the rule picks out the world that's behind, and not any world with a counter. The second group is Frost going away, as a world does once Consul drops it. Both worlds have an up of one, and Frost's ends after twenty seconds with stale, which is how Prometheus marks a series that's gone, as it does when a target leaves. Two minutes in, world down has to fire for Frost, and for Frost alone. With up at zero from before, this test fails, as no alert fires. In compose dot y m l, the alerts service runs promtool's test rules from Prometheus's image, on the monitoring folder, mounted where Prometheus keeps its config. It's in the test profile, like our other tests, so that compose up leaves it out. CI runs it after the tests against MySQL and Redis. And the README gets a section on the alerts: what each one means, where they go, and how a real mail server would take Mailpit's place.
15:25 Docker compose run dash dash rm alerts runs both test groups, and promtool reports success. There's nothing to build first, as the alerts service runs Prometheus's image as it is.
Wrap-up
15:38 So, today Prometheus got two alerting rules, for what players would notice: a world that runs fewer than nineteen ticks a second, and a world that's down, or gone from the targets. Each alert goes pending, and fires once its rule has held for a minute. Up at zero let go of a crashed world once Consul dropped it, and it would miss a world that stops, so the rule looks back over a day instead. Alertmanager emails us when an alert fires and when it resolves, with the alerts of one name in one email, to Mailpit, in Docker, which we read with curl and jq. And promtool tests both rules in CI, with the world that left among the cases.
Playing it
17:14 Let me know in the comments what you'd like to see next. And if you want to follow along with the series, subscribe to the channel. I hope you find this useful, and thanks for watching.