C++ sanitizers, fuzzing and coverage in Docker

Intro

0:00 Hello and welcome back to episode twenty-eight, where we're going to look for the bugs that our tests don't find, with sanitizers, fuzzing, clang-tidy and a coverage report, and fix what they turn up. Let's begin with a quick look at where we're going to be by the end of the episode.

0:15 The coverage report lists each file of our code, with how much of it the tests ran, and CI publishes it next to the API docs. Inside a file, each line shows how many times the tests ran it, and the lines that no test reached are in red.

0:32 On the website of the series you can find more information about the project. The code for the project is open source on GitHub, tagged at each step of each episode so that you can follow along.

Last time

0:44 Now, before we get to today's work, let's quickly go over what we did last time. The items folder became a content folder, with a library that the world servers, the clients and the bots read the game's content with. The kinds of monsters and NPCs, and where they stand, moved into it, out of C++ tables. Monsters and NPCs go to the players by their kind now, which halved what the NPCs cost on the wire. Each world lists a hash of its content in Consul, so that a client stays off a world whose content it doesn't have. And the Ash Skeletons joined the Ashlands as an entry in a file.

Today

1:22 Our tests check the cases we thought to check. However, a C++ program can read past the end of an array, or two threads can write one variable at once, and a test still passes, as nothing crashed that time. So today, we're going to build the game with the sanitizers, which add checks to the code as the compiler builds it, for memory errors, leaks and undefined behaviour. ThreadSanitizer is the one for data races, and we're going to run the servers with both while two hundred bots play. Then fuzzing, which hands a function input after input that nobody wrote by hand, clang-tidy, which reads the code for bugs that compile, and a coverage report, to see which code our tests don't reach. Each of them goes into CI, and what they find, we're going to fix in this episode. All right, let's get started with the sanitizers.

Sanitizers

2:12 AddressSanitizer comes with Clang and GCC, as a part of the compiler that adds a check around each read and write of memory. A runtime library goes with it, and keeps track of what's allocated and what's been freed. So it catches a read or a write out of bounds, or after a free, at the moment it happens. Its documentation puts the typical slowdown at about two times, which is fine for tests. On Linux, LeakSanitizer comes with it, and looks for leaks as the program exits, which is memory that nothing points to any more. And next to them, UndefinedBehaviorSanitizer, or UBSan, catches what C++ leaves undefined, such as a signed integer that overflows, or a shift wider than its type.

2:55 In the top CMake file, we start with a line that the build with Clang is going to need. We turn off CMake's scan for modules, and its comment says why: with Clang, CMake looks for C++20 modules, which we don't use, with a tool we'd have to install. Then the sanitizers, as a cache variable, nanommo sanitize, which is a string we set on the command line, and empty by default. Its comment lists what it takes: address comma undefined, or thread, which can't go with address, as ThreadSanitizer lays out its own record of memory in a way that clashes with AddressSanitizer's. When it's set, each file gets dash f sanitize with that list, and dash f no sanitize recover. So undefined behaviour stops the program at the first case, as a memory error does, and a test fails on it instead of printing a warning and carrying on. The link gets dash f sanitize too, which brings in the sanitizers' runtime libraries. These lines sit above FetchContent, so that abseil, protobuf and the other libraries are built with the sanitizers as well, and what our code hands them is checked too. In the Dockerfile, the sanitizers get stages that build with Clang. GCC has sanitizers too, but GCC 13 can't compile abseil with UBSan on, as it stops at a constant expression it fails to evaluate. So we're going to use Clang, which also comes with the other tools we need today. The clang stage starts from our toolchain stage, with CMake, Ninja and the libraries. Its comment says what took a while to find out: on Linux, Clang links GCC's runtime library unless told otherwise, and the sanitizers need some of Clang's own. It installs Clang 18, which is the version that Ubuntu 24.04 comes with, and lib clang R T, with the sanitizers' runtime libraries and Clang's own runtime. Then LLVM 18's tools, such as the one that turns the addresses in a sanitizer's report into functions and lines. LLVM's folder goes first on the path, so that those tools go by their plain names, and C C and C X X tell CMake to build with Clang. The linker flags pick compiler R T, Clang's runtime library, as UBSan checks 128-bit multiplications with a function from it that GCC's doesn't have. And GCC's unwinder stays, for the exceptions that some of the libraries throw. The sanitize stage takes the sanitizers as a build argument, with address comma undefined as its default, and copies the source in. It builds R E L with deb info, which keeps the optimisations and adds the line numbers that the reports need, and then it runs all of the tests, like the test stage.

5:31 Let's build the sanitize stage, and run the tests in it. Dash dash progress plain prints the build's output as plain lines, each with Docker's step number in front. The first grep keeps those, as Docker prints the end of a failed step a second time. The second keeps each leak, with the four lines of its stack, and the summary of the tests, and sed takes the step numbers off. Two of the client's tests fail, and both of them load the skeleton's model. Each test runs in a process of its own, and in each, LeakSanitizer reports two blocks of memory that nothing frees. raylib allocates them in load g l t f, as it loads the model. In raylib's code, load g l t f gives a model with bones its current pose, and a matrix for each bone. Unload model frees the meshes, the materials, the bones and the bind pose, but not those two arrays. The players' characters would leak them too, but Mixamo's files stay out of the repo, so in Docker, the tests load the skeleton alone.

6:34 In characters dot c c, we're going to free them ourselves, as the client is where we unload the models. Unload with bones, in the unnamed namespace, takes the model, and unloads it with raylib. Then it frees the current pose and the bone matrices with mem free, raylib's counterpart of the allocator it used for them. Its comment says why it's there: raylib six point zero's unload model leaves a model's pose and bone matrices behind. Raylib's main branch frees them, and so does six point two, from October 2026. So when we move to it, this function goes, as freeing them twice would be a crash, which AddressSanitizer would point at. The characters call it in their two places: where a model without the clips we need is let go, and in the destructor.

7:23 We build the stage again, with dash dash no cache filter, so that Docker runs its tests again instead of reusing a cached result. All of the tests pass, the client's too, with AddressSanitizer, UBSan and LeakSanitizer watching each of them.

Under the bots

7:39 The tests reach a part of the code, and the bots reach a lot more of it, the way players would, so we're going to run the servers with the sanitizers while the bots play. Compose reads override files on top of compose dot yaml, like the profile file from episode eight, so compose dot sanitize dot yaml changes five services. Their build is shared, so it goes in an extension field, x dash sanitize, which Compose leaves alone, with a YAML anchor that each service merges in. It builds the sanitize stage, with the sanitizers from the SANITIZE variable in the shell, and address comma undefined when it's not set. The login server merges it in, and gets an image name of its own, nanommo login sanitize. So the plain image stays as it is, and the next plain build doesn't pick up this one. The sanitize stage has no entry point of its own, so we point it at the login server's binary in its build folder. The world server gets those lines too, with its own names, and so does the saver, which takes the players from Redis to MySQL, and gets checked as it does it. The bots get them too, so that their code is checked as they play, and the tests, which run against MySQL, Redis and Consul, get the sanitize stage as well.

8:53 Compose file is an environment variable that lists the files each compose command reads, so we set it once, to compose dot yaml and the sanitize file after it. We bring up the login server, the world server and the saver, built with AddressSanitizer and UBSan, and wait for them to run. Then two hundred bots, built with them too, join over thirty seconds. We let them play for two minutes: they walk, get chased by the skeletons, go through the gates, chat, and ask the storyteller for stories. The storyteller isn't running here, so each request ends in an error, which is a path through the server's code worth checking too. The world server's log says that two hundred players joined. Then we stop them, as LeakSanitizer looks for leaks when a program exits, and the servers' logs have no reports from the sanitizers, and no runtime errors from UBSan.

9:42 ThreadSanitizer is the sanitizer for data races: two threads touching one piece of memory, at least one of them writing, with nothing between them that orders the two, such as a lock or an atomic. It slows the code down about five to fifteen times, and takes five to ten times the memory, and it can't go with AddressSanitizer, so it gets a build of its own. It sees each thread start and each lock, and each atomic in the code it built. And it keeps a record of which thread touched each part of memory last, at addresses of its own.

10:13 Let's build the server's tests with SANITIZE at thread, in a container from the clang stage, with the source mounted and the build in a volume, so that the next commands can run it. We run the Consul test, against a Consul from our compose file, and the process dies of a segmentation fault as the test starts, with no report from ThreadSanitizer. So we run it again under gdb, the GNU debugger, which prints the stack of the thread that crashed. The crash is inside ThreadSanitizer's own allocator, as a thread allocates memory, and the bottom of the stack says whose thread it is: glibc's handle requests, which answers get adder info A. httplib, which we talk to Consul with, looks names up with get adder info A, the asynchronous version, so that a slow answer can't hold it past its timeout. glibc answers on a thread of its own, which ThreadSanitizer didn't see start, so it has no record of that thread when it allocates, and crashes instead of reporting.

11:08 httplib comes with a switch for it, in its CMake options, next to the ones we set for it in episode twenty-one. When the list of sanitizers has thread in it, which CMake's matches finds anywhere in the list, as it's a regular expression, we turn httplib's non-blocking get adder info off. Then httplib looks names up with plain get adder info, which waits for the answer on the calling thread. Its comment says why: glibc looks names up for httplib on a thread ThreadSanitizer can't see. The other builds keep the timeout, as our Consul thread and the storyteller's threads look names up while the server runs. Running the client's tests under ThreadSanitizer a few times turns up a second thing it can't follow. In my runs, about one run in four reported a race inside Mesa, the driver that draws the client's tests in Docker, on the CPU, between its own threads. Mesa orders those threads in code that isn't built with the sanitizer, so ThreadSanitizer sees the two accesses without what's between them. None of our code is on those stacks, and the client's code runs on a single thread. So t san dot supp is a suppressions file, and called from lib gallium tells ThreadSanitizer to leave out what Mesa's library does with threads. Its comment says that Mesa draws the client's tests in Docker, and orders its threads in code ThreadSanitizer can't follow. In the sanitize stage, t san options points ThreadSanitizer at the file, and the other sanitizers leave the variable alone.

12:41 This time, SANITIZE is thread, so Compose builds the servers and the bots with ThreadSanitizer, and the bots play for two minutes again. We stop them, and grep the world server's log for ThreadSanitizer's warnings, with the two lines after each, and for the line of the earlier access that it races with. Sed takes the service's name off the front of each line, and the binary's name and offset off the end. There's one data race, as the server stops. The main thread writes the stop flag in m g stop, as it shuts down the web server that serves our metrics, which is civetweb, inside prometheus-cpp. And the master thread of that web server read the flag earlier, in master thread run, with nothing between the two. The bots report one warning too, as they serve their metrics with prometheus-cpp as well. civetweb's stop flag is a plain int, which its threads read in a loop while another thread sets it. In C and C++, that's a data race, which is undefined behaviour, even if it tends to work on today's processors.

13:43 civetweb has a build option, stop flag needs lock, that reads and writes the flag atomically. In April 2025, its developers fixed the last plain read in that path, to make ThreadSanitizer happy, as the commit says. Assigning the flag compares and swaps it atomically, with what it read, and the fix reads it with an atomic load too. However, civetweb's last release is one point sixteen, from 2023, which is the one prometheus-cpp comes with, so the fix isn't in our copy.

14:15 So we're going to do both, in the top CMake file. FetchContent can patch what it fetches, with a patch command that it runs once in the source it unpacked. So prometheus-cpp gets patch, with dash p one, which drops the first folder of each path, and dash i with our patch file. Then, once prometheus-cpp is available, its civetweb target gets the define, stop flag needs lock. Its comment says what we're asking for: civetweb serves the metrics, and stops its threads with a plain int unless we ask for atomics. We turn it on in each build, whatever the sanitizers, as a data race is undefined behaviour whether or not a tool is watching. The patch goes in a patches folder, as civetweb stop flag dot patch. Its first lines say what it is, and patch skips them: civetweb's fix for a data race on its stop flag, from its main branch, which no release of it has yet. The rest is the commit's change to civetweb dot c, against the copy in prometheus-cpp's third party folder. When prometheus-cpp comes with a civetweb that has it, the patch goes, and patch would refuse to apply it twice.

15:26 With SANITIZE at thread again, we build the servers with ThreadSanitizer, and the bots play for two minutes. We stop them, and count ThreadSanitizer's warnings in the logs of the servers and the bots, which is zero. So, between the Consul thread, the storyteller's sixteen threads, the threads of the metrics server and the game loop, ThreadSanitizer finds no race while two hundred players play.

Fuzzing

15:50 Tests and sanitizers run the code with the inputs we give it. Fuzzing makes up the inputs: libFuzzer, which comes with Clang, calls a function of ours with input after input, and watches which branches of the code each input reaches. An input that reaches code that the earlier ones didn't is kept, and the next inputs are made from those, by flipping bits, cutting and joining them. So it works its way into the code, much faster than random bytes would. The function is a fuzz target, LLVM fuzzer test one input, which takes the bytes and their size, does something with them, and returns zero. With the sanitizers on, a memory error or a broken check in our code stops it, and libFuzzer saves the input that did it.

16:33 The best targets for a fuzzer are functions that take input from outside. Two of ours do: the storyteller's lines, which take whatever the model writes, and the world, which takes whatever a client sends. In the top CMake file, an option, nanommo fuzz, builds the fuzzers in the fuzz folder, with Clang, as libFuzzer is a part of Clang. It's off by default, as GCC, which our plain build uses, can't build them. When it's on, our code gets dash f sanitize fuzzer no link, which adds the counters that libFuzzer reads, without linking libFuzzer's main function into each of our programs. The comment says so: so that libFuzzer sees which of our code each input reaches. It goes after FetchContent, so that the libraries' code doesn't count, as an input that reaches deeper into protobuf's parser isn't what we're after here. Further down, the fuzz folder is added after the client, when the option is on. In the fuzz folder's CMake file, a loop makes an executable for each fuzzer, stories and world, which links the server's core library, with the code we're fuzzing in it. Each gets dash f sanitize fuzzer at the link, which brings in libFuzzer's main, so that the executable runs the fuzzing loop around our function. And the world fuzzer gets the path of the content folder as a define, so that it plays with our real monsters and weapons. Stories dot c c is the first fuzz target. It includes stories dot h first, as the header of the code it fuzzes, and then the standard headers for sizes and strings. Then Abseil's check, for checks that stop the program when they fail, and chat dot h, for the most a line of chat can be. The target is extern C, so that libFuzzer finds it by its plain name, and it takes the bytes as a story, in a string view, without copying them. Its comment says why: the model can write anything, and each line has to fit in the chat. Any bytes at all are a story it could write, as the model picks tokens, and a small model picks odd ones now and then. So for each line that story lines makes of it, we check that the line is no longer than the most a line of chat can be. A check that fails aborts the program, which libFuzzer counts as a crash, and it saves the input that did it. World dot c c is the second, and it plays the game. It includes world dot h first, then the fuzzed data provider header, which comes with libFuzzer. Then bit, for bit cast, chrono for the length of a tick, c math for is finite, and the headers for sizes. Then Abseil's check, and our content, levels, monsters and terrain headers, for the island, its monsters, and the most health that a level gives. Its target takes the bytes like the first one. The comment over it says what it's for: a client can send any input, so the fuzzer plays with any input, and checks what the world has to keep true. Making the island takes a while, as the terrain comes from layers of noise, so each input plays on a copy of one island, which we make on the first call, as the comment says. It reads our content folder, for the weapons and the monsters, and makes the world with new, which isn't deleted. That's how Google's style guide makes a static that would otherwise run a destructor at exit. Then it spawns the monsters of the Meadows, like the server does as it starts. Fuzzed data provider comes with libFuzzer, and hands out numbers and booleans from the bytes, in order, so the bytes turn into a list of things to do. Any float, a lambda, takes thirty-two bits from the bytes and reads them as a float, with bit cast, from the bit header. Its comment says why: any thirty-two bits make a float, NaNs and infinities too, as a client could send. The provider's own floats are always finite, which would leave those out. Each input starts from a fresh copy of the island. While there are bytes left, we pick one of three players, and one of four things to do. A player who isn't playing joins, the way the server would let them in. Or a player sends any input at all, two of those floats for the direction, and two booleans for running and swinging. Or the world steps forward a tick of fifty milliseconds, or a player who's playing leaves. At the end, we check each player that's still in the world. They have to be somewhere real, with an x and a z that are finite numbers, with health of zero or more, and with no more health than the most that their level allows. A NaN that got past the server's check would push a player off the map, and a bug in a fight could heal a player past their most. These checks run at the end of each game, so a broken world turns into a crash, with the input that made it saved next to it. The Dockerfile gets a fuzz stage, from the clang stage, which builds the two fuzzers with AddressSanitizer and UBSan on, so that a memory error or undefined behaviour counts as a crash too. It builds those two targets and nothing more, as fuzzing needs none of the programs, which keeps the stage quick. And the lint stage checks the fuzz folder, like the rest of our code.

21:31 We build the fuzz stage, as an image called nanommo fuzz, and run the stories fuzzer, with dash seed one, so that each run makes its inputs in one fixed order. Within seconds, it stops on our check: story lines made a line longer than two hundred bytes, and libFuzzer saves the input that did it to a file. Story lines breaks a story between words, so the one way to get a line that long is a word longer than a line, which went on a line of its own, whole. The server's clean chat drops a line that long, so that part of the story is lost, and a story of nothing but such a word reaches the player as nothing at all. A small model can write a long run of dots or letters like that. The world fuzzer gets a minute, and finishes with no crash, so the world held up through the games it played.

22:20 In stories dot h, the comment on story lines says what's going to happen to such a word: a word longer than a line gets lines of its own, cut between two of its characters. In stories dot c c, cut finds where to cut a word that's longer than a line. It starts from the most a line can be. A character past ASCII is two to four bytes in UTF-8, and protobuf refuses a string field that isn't valid UTF-8, so the client would drop the whole message of chat. So the cut goes back while the byte there is in the middle of a character. As the comment says, in UTF-8, each byte after a character's first starts with the bits one zero. If it went all the way back, the word isn't UTF-8 at all, and any cut will do, so it cuts where the line ends. In story lines, while the word is longer than a line, the line so far goes out first, and then a line of the word, up to the cut. The rest of the word carries on like any other word. The word is a string view, so remove prefix moves its start along, without a copy. The test cuts a word of é, which is two bytes, after an A, so that the first cut can't be at two hundred, which is in the middle of one. We check that the words before it get a line of their own, that the first cut is at a hundred and ninety-nine, and that the two lines make up the word again. And the fuzzer checks the cut too, with utf8 range, the UTF-8 check that comes with protobuf. If the story is valid UTF-8, each line has to be. Its comment says why: protobuf refuses text that isn't UTF-8, so a line mustn't end inside a character. A story that isn't UTF-8 to start with can be cut anywhere, as it's broken already. The stories fuzzer links it, from protobuf's third party folder, where protobuf keeps it for checking strings as it parses them. We include it with its folder's name, utf8 range slash utf8 validity dot h, which our lint asks for, so its folder's parent goes on the include path.

24:22 The stories' tests pass, the new one too. And the stories fuzzer runs for a minute with nothing to report, so in all of those inputs, no line came out too long or ended inside a character.

clang-tidy

24:41 clang-tidy reads the code with Clang's parser, the way the compiler does, and runs checks on it: for likely bugs, for code that's slower than it needs to be, and for style. It comes with Clang's static analyzer too, which follows the paths through a function looking for a null pointer read, a leak or a use of something uninitialised, without running it. So it finds bugs that compile, some of which the compiler's warnings miss.

25:05 Dot clang tidy, at the top of the repo, says which checks to run, for each file under it, and clang-tidy finds it the way clang-format finds dot clang format. We start from no checks at all, and turn on whole families of them, with a few left out, as they flag what we mean to do. Bugprone is the family for likely bugs, such as a moved-from object that's used again, or an optional read without a check. Easily swappable parameters flags two parameters of one type next to each other, which is each x and z we pass. Exception escape flags main and our handlers, for exceptions that the libraries might throw, and we don't use exceptions. Implicit widening flags sizes multiplied in int, which stay far below what an int holds here, and narrowing conversions flags each int that turns into a float, which the client draws with. Clang analyzer is the static analyzer, and concurrency is the family for threads. Its M T unsafe check flags get env, which isn't safe while another thread changes the environment, but nothing in our code changes it. Performance is the family for needless copies and allocations. Avoid endl flags s t d endl, which flushes each line of a server's log, so that Docker shows each line as it happens. And enum size flags enums that would fit in a byte, which saves us nothing worth a change. Two of the checks get options of their own. No automatic move flags a const local that's returned, as const stops the move, so it's copied. Abseil's status is allowed, as copying one costs no more than a reference count. Unused return value would flag each error code that Asio returns from the calls that also set the one we pass, so error codes come off its list of types to check. Each finding is an error, so the stage fails on it. The header filter checks our own headers, and not the libraries' or the ones that protoc writes, whose names have a dot before the h. The tidy stage installs clang-tidy 18, from Ubuntu, on top of the clang stage, and copies the source in. It configures with nanommo fuzz on, so that the fuzzers are in the compile commands too. clang-tidy reads the headers that protoc writes, so those come first, as the comment says, and building the protocol and the content libraries makes them. Then run clang-tidy runs clang-tidy on each file in the compile commands that CMake exports, a few at a time, for our folders. It leaves out the tests, as clang-tidy 18 doesn't see that gtest's assert true checks an optional before we read it, and our tests do that a lot.

27:41 We build the tidy stage, and grep its errors, from the lines with Docker's step numbers, as Docker prints the end of a failed step twice. Sed takes the source folder off each path, and the name of the check off the end. Three optionals are read without a check that clang-tidy can follow: in the bot, on the login screen, and in the server's talk. Each is fine today, as something checks or sets them a step earlier, but nothing says so where they're read. The broadcast pushes each piece of a cell's world into a vector that grows one push at a time, and the send of a datagram copies the shared pointer to its bytes. A move of a monster copies anyway, a path is built from temporary strings, the token is copied as it's returned, and the world fuzzer's switch has no default.

28:28 In the bot, asking for a story is set together with the storyteller it's found, and reset with it, so the bot asks the storyteller when it's asking and it knows a storyteller. The check is there where it's read, so that a later change that sets one without the other can't read an empty optional. The login screen builds its login window when it has a world, and the world's name goes in the window's title. So build login takes the title as a parameter, and its caller makes the title, from the world it has just checked, with str cat, like before. A reference to the world could dangle, as going back to the list of worlds resets it, while the title is a string of its own. In the server's talk, the prompt comes from the words, and we refuse the story when either of them is missing, as we print the words further down, so they have to be there. In the broadcast, the pieces get their room before we push them, as we know how many there are: the players, the monsters, and the NPCs once a second. That's a vector for each cell in each tick, so it saves a few allocations. In send to, the handler takes the shared pointer by moving it in, instead of copying it, as we're done with ours. A copy of a shared pointer counts up and down atomically, once for each datagram we send, so the move saves that too. The handler still keeps the bytes alive until the datagram has gone. In the world, a monster is plain data, so moving it copies anyway, and the move goes. The characters include Abseil's str cat, and build each path with it, which works out the size once, and writes the string in one allocation. The token isn't const any more, so returning it moves it, and the world fuzzer's last case is the default.

30:08 We build the tidy stage again, and it finds nothing, so once it's in CI, a change that brings one of those back is going to fail.

Coverage

30:17 Last, we're going to see which of our code the tests run. Clang's source-based coverage counts how many times each region of the code runs, from the compiler's own map of the source, so it's accurate to the line and the branch. It takes three steps: we build with the counters, run the program, which writes its counts to a raw profile, and then LLVM's profdata merges the profiles, and LLVM's cov turns them into a report.

30:42 In the top CMake file, an option, nanommo coverage, counts the lines the tests run, with Clang. Our code is compiled with dash f profile instr generate, which adds the counters, and dash f coverage mapping, which maps them back to the source. The link adds the runtime that writes the counts out as the program exits. It all goes after FetchContent, next to our warnings, so that it counts our code, and not the libraries'. The coverage stage builds with the option on, from the clang stage, as the counters and the tools that read them are Clang's and LLVM's. It builds everything, the programs too, which the report is going to need, and it doesn't run the tests yet, as some of them need MySQL, Redis and Consul. Coverage dot s h runs the tests and writes the report, as a shell script that stops at the first error. LLVM profile file says where each process writes its counts. The tests run one process each, so percent p puts the process's id in the name, and each test's process writes what it ran to a file of its own, as the comment says. We run all of the tests, the client's too, on a virtual display, and LLVM profdata merges the raw files into one profile. The arguments the report needs go into the shell's positional parameters, which is the list a shell script has for its own arguments, so that each stays a single word. They're the profile, and a pattern for the files to leave out, which are the libraries, the tests and protoc's code. The pattern is a regular expression on the files' paths. Then a loop goes over each of our binaries, the ones whose names start with nanommo, and each of them gets dash object. That's the programs too, as the comment says, so that code no test reaches counts, at zero. A report from the test binaries alone leaves out the files that no test links, such as the server's game loop, and makes the total look better than it is. LLVM cov show writes the HTML report to slash coverage, with the counts on each line. Report prints a table of each file to the log, and export writes the totals as JSON, next to the report. In compose dot yaml, a coverage service builds the coverage stage and runs the script. It gets the hosts of MySQL, Redis and Consul in its environment, as our code reads them from there, and it mounts site slash coverage, so that the report lands next to the API docs. It waits for Consul and Redis to be healthy, and for Flyway to make the tables, and like the tests, it's in the test profile, so that a plain compose up leaves it out.

33:15 We run the coverage service, and keep a few rows of its table, and awk prints the lines each file has, and the share of them the tests ran. Over our code, the tests run about forty-two percent of the lines. The world is mostly covered, and so is story lines, while the server's game loop, its sessions and the bot are at zero, as no test runs a server.

33:36 The report is a page for each file. Server dot c c is at zero, as the tests use the world and the broadcast's pieces, but don't start a server. Inside it, each line of the tick shows a count of zero, in red. In a file the tests reach, each line shows its count, and the parts of a line that didn't run are marked in red, so a red mark is a case that no test tried.

In CI

34:00 The checks go into CI, so that each push runs them. The build job runs clang-tidy after the tests, so that a finding fails the push. And it runs the coverage service after the API docs, so that the report goes up to GitHub Pages with them, in site slash coverage. A new job, sanitize, runs on Ubuntu, next to the build job, and it runs twice, with a matrix of the two lists of sanitizers, address comma undefined, and thread. Each run gets its list in SANITIZE, and the compose file variable, so that each compose command uses our override. The first step after the checkout leaves ThreadSanitizer room for its memory. GitHub's runners randomise where things go in memory with thirty-two bits of address, while ThreadSanitizer's layout fits up to twenty-eight. Docker Desktop's kernel uses eighteen, which is why we didn't need it on the Mac. So we set the runner's kernel to twenty-eight bits, with sysctl. Then it builds the sanitize stage, which runs the tests, with SANITIZE from the job's environment, and runs the tests against MySQL, Redis and Consul with the sanitizers on. The last step has fifty bots play for a minute against the servers, all built with the sanitizers. Compose stops them, so that the checks that run at exit run too, and the job fails if a sanitizer printed anything in the logs of the servers or the bots. AddressSanitizer would have stopped a server, while ThreadSanitizer reports and carries on. A third job, fuzz, runs next to the others, on a runner of its own. It checks out the code, and builds the fuzz stage as an image, nanommo fuzz, so that it can run the fuzzers from it. Then it gives each fuzzer a minute, the stories first and then the world. A minute isn't long, but it can catch a change that breaks what the fuzzers check.

35:50 Last, the README gets a section on checking the code, after the code style. It has the sanitizers and what each of them catches, the stages that run them, and the address randomisation that ThreadSanitizer needs. Then playing against servers built with the sanitizers, the fuzzers and what they check, clang-tidy's choice of checks, the coverage report, and the patches folder.

Wrap-up

36:15 So, today we built the game with AddressSanitizer, UBSan and LeakSanitizer, which found raylib leaving two arrays behind for each model with bones, and the servers ran clean while the bots played. ThreadSanitizer found a data race in civetweb's stop flag, which we fixed with its own option and a patch from its main branch. That was after we kept it away from what it can't follow: glibc's lookup thread, and Mesa's own ordering. The stories fuzzer found that a long word lost a story, and the world fuzzer played through a minute of games without a crash. clang-tidy found a few optionals read on trust and a few needless copies, and the coverage report shows that the tests run about forty-two percent of our lines, with the server's loop at zero. Each of them runs in CI from now on.

Playing it

37:57 Let me know in the comments what you'd like to see next. And if you want to follow along with the series, subscribe to the channel. I hope you find this useful, and thanks for watching.