Integration testing a C++ game server

Intro

0:00 Hello and welcome back to episode twenty-nine, where we're going to write a test that starts our world server and plays it, with players over TLS, so that the code our coverage report had at zero gets tested. Let's begin with a quick look at where we're going to be by the end of the episode.

0:17 The coverage report has counts for the world server's code now, as a test starts a server, and players join it, see each other and leave. Inside server dot c c, the lines of the tick ran a few times for each test, as each test stops waiting as soon as what it checks has happened.

0:35 On the website of the series you can find more information about the project. The code for the project is open source on GitHub, tagged at each step of each episode so that you can follow along.

Last time

0:47 Now, before we get to today's work, let's quickly go over what we did last time. We built the game with the sanitisers, and LeakSanitizer found raylib leaving two arrays behind for each model with bones. ThreadSanitizer found a data race in the stop flag of CivetWeb, the web server that serves our metrics, while the bots played. A fuzzer found that a word longer than a line lost the storyteller's story, and clang-tidy found a few optionals read on trust and a few needless copies. And the coverage report showed that the tests run about forty-two percent of our lines, with the world server's game loop, its sessions and the bot at zero.

Today

1:27 The code that each player of the game goes through, from the hello to the broadcast of the world, is code that no test runs. The bots play against it in Compose, and the sanitisers watched them last time, but a change that stopped players from seeing each other would still pass CI. So today, we're going to start a world server inside the server's tests, in the test's own process. That way, the test can look at the server's state, the coverage counters count its code, and the sanitisers check it along with the rest of the tests. The server is going to listen on port zero, which has the system pick a free port, so that the test can't clash with a server that's running. Players connect to it over TLS, the way the client does, and we check that a player joins, that two players see each other, and that a player who leaves goes from the other's world. A second test checks that an account can't play twice at once, and then we run the coverage report again, to see how much of the server the tests reach. All right, let's get started with the port that the server listens on.

Port zero

2:27 A test that starts the server on port zero has to find out which port the system picked, so that its players can connect to it. So in server dot h, the server gets a function, port, which is const, as it reads the port without changing anything. Its comment says what it returns: the port it listens on, over TCP and UDP. Given port zero, Create has the system pick a free one. In server dot c c, port asks the acceptor for its local endpoint, which is the address and the port it's bound to. The guide doesn't use exceptions, so we use Asio's overload that takes an error code, which can't fail on an acceptor that's open. Start used to bind UDP to the number it was given, and with zero, UDP would have got a second free port of its own. However, the clients send their UDP hello to the port they reach over TCP, so UDP has to take the port that TCP got. Start's parameter is called port too, which hides the function inside Start, so we call the function through this arrow port. The comment over it says why: on port zero, the system gave TCP a free port, and UDP takes that one too. The game's main refuses port zero, so for the game, nothing changes.

A server in a test

3:40 In the server's CMake file, server test dot c c joins the server's tests. The server's tests depend on the step of the build that makes the protocol tests' certificates, from an authority of our own, and the comment over it says why: the server's test talks TLS with them. The folder they're in goes to the test as a define, nanommo test TLS, in the protocol's build folder, which is how the protocol tests find it too. The build makes those certificates with our certificates script, so the test doesn't need the ones the servers use in Compose. Server test dot c c includes server dot h first, as the header of the code it tests. We start with the player that the tests are going to need, so the headers are the ones it uses. From the standard library, algorithm, for any of, array for a buffer, cstddef and cstdint for sizes and the port, optional, and string. System error is for the error codes that Asio hands our handlers, and utility is for move. From Asio, buffer, and address, for this machine's address, then host name verification, which checks the name on the server's certificate, stream base, for the side of the handshake, and write. Test player is a player, the way the client is one: over TLS, with a token, and it keeps the last world the server sent, as its comment says. The bot is a client too, but it walks off at random, and it doesn't keep who it sees, so the test gets a player of its own, which stands where it joins and remembers the world. Its constructor takes the I O context to run on, the client's side of TLS, the port, and the token. It checks that the server's certificate is for localhost, and connects to one twenty-seven dot zero dot zero dot one, this machine's loopback. Each step is asynchronous, so the constructor starts the connection and returns straight away, and the connection goes ahead as the test runs the I O context, which the server runs on too. If the connection fails, the player is closed, which a test can check. Once it's connected, it shakes hands over TLS, as the client, and then greets the server. Sees looks through the players of the last world for an id, with any of, from the standard library's ranges. Its comment says what it answers: whether the last world the server sent has the player in it. It goes by the player's id, as the worlds that the server sends between full ones leave the names out, and keep the ids. World returns that world, for a test to look at, and closed says whether the connection is closed, which is how a test sees that the server refused a player. Greet first checks how the handshake went, and one that failed, such as for a certificate that isn't for localhost, closes the player. Then it sends the hello with the token, in a frame the way the server reads it, and the frame stays in a member while the write goes out. The write's handler leaves errors alone, as a write that fails breaks the read too. Read reads what the server sends, and a read that fails is how Asio tells us that the connection has closed, so the player is closed then. The bytes go into the frame reader from episode three, which hands back each whole message, as each one the server sends a player is a world. Then we parse the message as a world, and check that it has no chat in it. The comment over that check says why: chat comes in worlds of its own, with nobody in them, so a line of chat would make the player look alone for a moment. A world that passes becomes the last world that the player keeps. Its members are the TLS stream, the hello, a buffer of four kilobytes for each read, the frame reader, the last world, and whether it's closed.

7:14 The fixture is going to run the server on the test's thread, next to the test players, which works because of how Asio runs what's waiting. Asio's I O context keeps the handlers that are ready to run, which are the functions we gave it to call once something has happened, such as a connection, bytes to read, or a timer that fired. The server puts its tick there, on a timer that fires twenty times a second, and its sessions put their reads and writes there. Our test players put their connects, handshakes and reads there too. In the game, the server's main runs the I O context, and here the test does, one handler at a time. Between two handlers, nothing else runs, so the test can ask the server how many players it has, or check what alice sees, with no data race, as no other thread touches them. A blocking call would stop the whole queue, as a player waiting for the server's answer would keep the server from running to give it. And when what the test waits for doesn't happen within five seconds, it stops waiting, and the test fails, instead of hanging.

8:16 Back in server test dot c c, the fixture needs more headers. Chrono is for the length of a tick and the deadline, functional for the function that the tests pass to run until, memory for unique pointers, and vector. From Abseil, status or, and str cat, which builds the paths of the certificates and a key in Redis. Then GoogleTest, our accounts, which the test makes its players' accounts with, and test services, for MySQL and Redis. K TLS is the folder with our authority's certificates, the protocol tests' ours folder, and its comment reads: the certificates the protocol tests made, with our own authority. Server test is the fixture, with what each of the server's tests needs. Set up gets MySQL and Redis, like the other tests that need them, and GTEST SKIP skips the test when they aren't set, so the test stage of our Dockerfile, which has neither, skips it. The server loads each player from MySQL and Redis as they join, and saves what changed to Redis on each tick, so the test uses the real ones, like the server in the game does. Alice gets an account, and then come the tokens and the players, on that MySQL and Redis. The server takes them by reference, so the fixture owns them for as long as the test runs. Create also wants a storyteller, and ours points at a host where nothing listens. Its comment says why that's fine: nobody asks it for a story, as the world has no NPCs. TLS comes from our authority's server certificate and key, and the client's side trusts our authority, so the players check the server's certificate the way the game's client does. Both return a status or, and assert true stops the test if either file can't be read. Then we create the server, on port zero, with ticks of fifty milliseconds, which is twenty a second like the game. Create returns the server, or why it couldn't start, such as a port that's taken, and the assert stops the test there, with the status in its message. Its world gets a random name, like the test's accounts, so that two runs don't share their players' keys in Redis. Tear down deletes what each test made, so that the database doesn't fill up with test players. For each account, it deletes the player's hash in Redis, and takes it out of the set of unsaved players, in a pipeline. The comment says why: the server saved the player in Redis, for the saver to take to MySQL. Without it, each run would leave a player in Redis for an account that's gone. Then it deletes the account, and the foreign key's cascade would take any rows of its player in MySQL with it. New account creates an account with a random name, from test name. Its password hash is the word hash, as the tests hand out tokens without a login, so nobody checks a password. Then it keeps the account's id for tear down, and returns it. Token issues a token for an account, which is what the login server hands a client after a login. A token lasts thirty seconds in Redis, and each test uses its tokens within a second or two. If Redis can't store one, expect true fails the test with the status, and the player gets an empty token. Run until is how the tests wait for the server, without sleeping for a time we'd have to guess. Its comment says what it does: it runs the server and the players until done says so, for five seconds at most. Run until is the loop from the drawing: run one until runs the next handler that's ready, and returns zero once the deadline has passed with nothing to run, and then run until returns false, and the test's assert fails. Then the members, which the tests reach directly, as a fixture's tests are classes that derive from it. First MySQL and Redis, the accounts, in a vector, so that a test can make as many as it needs, and alice's id. Then the tokens, the players and the storyteller, which are unique pointers, as set up makes them once it knows that MySQL and Redis are there. Then the world's name, and the content, which is empty, with no items, monsters or NPCs, as its comment says, so nothing in the world moves but the players. Then the metrics, which the server counts its ticks and messages in, the way it does in the game. TLS for both sides is in status or members that set up fills in, and then come the server, and the I O context that the server and the players run on. The first test is that a player joins, which goes through the hello, the token in Redis, loading the player, and the first broadcast. Alice connects with a token from the fixture, and we run until alice sees alice, which is the server putting alice in the world that it sends to alice. Assert true stops the test there if that doesn't come within five seconds. Then we check that the world says that it's alice's, and that the server counts one player.

12:58 Let's build the tests and run them in Compose, next to MySQL and Redis. What comes after the service's name replaces its command, so we run the server's tests with dash dash gtest filter, which picks the tests whose names start with server, and grep keeps the lines that say how each went. The test passes: the server's log says that the player joined, and the player got a world with themself in it.

13:21 We built the sanitisers last time for code like this, so let's run the new test with them too. Compose file lists our sanitize file after compose dot yaml, so that the tests come from the sanitize stage, with AddressSanitizer and UBSan. U B San options with print stack trace has UBSan print the stack of what it finds. Then grep keeps the report, and the frames of the stack in our code and in Asio's timer, and sed shortens them: it takes off the frames' numbers and addresses, the paths and the template arguments. UBSan stops the test as the fixture is destroyed: a member call reached an object that isn't a deadline timer service, which is the service behind Asio's timers. UBSan checks each call on an object with virtual functions against the object's type, and the memory at that address doesn't hold a service any more. Read from the bottom, the stack says how we got there. The fixture's destructor destroys the server, the server destroys alice's session, and the session's TLS stream destroys a timer that Asio's TLS keeps for its reads and writes. The timer then calls its service, which belongs to the I O context. A class's members are destroyed in the reverse order of their declaration, and we declared the I O context last, so it went first, with its services. The plain build ran past it: a call through freed memory is undefined behaviour, which can crash, or seem to work, as it did here.

14:47 The I O context sits at the end of the fixture's members, so it's the first one to go, while the server that runs on it still has sockets and timers to close. So it moves to the top of the members, and its comment says why it's first: so that it's destroyed last, as the server's sockets and timers need it while they close. Then its old line at the bottom of the members goes, where it was the first member to be destroyed.

15:11 With the sanitize file in the compose file again, we run the server's tests, and with the I O context first, the test passes with AddressSanitizer and UBSan watching, and LeakSanitizer finds nothing to report as the process exits.

Seeing each other

15:25 A player who sees themself is a start, but the point of the server is that players see each other, so the test gets a second player, bob, who's going to leave too. A test player leaves by closing its connection, the way a client that quits does, so leave closes the socket under the TLS stream, with an error code that we don't need. Set up gives bob an account too, and the fixture keeps bob's id next to alice's. The test is named for what it checks now: players see each other until one of them leaves. Alice joins and sees alice, like before, and the server counts one player. Then bob joins, and we run until alice's world has bob in it, and bob's has alice. The broadcast sends each player the players near them, and both stand where new players start, so each sees the other, and the server counts two players. Then bob leaves, and we run until the server counts one player, and alice's world doesn't have bob in it any more. Both have to hold, as the server takes bob out when it reads the closed connection, and alice sees it in the next world that the server sends.

16:31 We run the server's tests again, in Compose, with the plain build. The log has two players joining and one of them leaving, and the test passes in a fraction of a second, as it waits for each check no longer than the server takes to get there.

One at a time

16:44 The server refuses a hello for an account that's already playing, so that one account can't have two characters in the world at once. That rule lives in the server's hello, and nothing tested it, so a second test pins it. An account plays once at a time: alice joins and sees alice. Then alice connects again, with a new token, while alice plays. We run until the server has closed that connection, and the server still counts one player. A test that something is refused should also show that the right case is accepted, so that it fails for the reason it names. So alice leaves, and we run until the server counts nobody. Then alice connects a third time, with a new token, and gets in. That shows that the second connection was refused for playing twice, and not for its token or the way it connected. The README gets a paragraph on the server's test, next to how to run the tests that need MySQL and Redis: what it starts, what it checks, and how it waits.

17:42 Both of the server's tests pass, and between them, the server's log has the refusal, which says that the token is unknown or the account is already playing. CI runs these tests on each push to main and each pull request, in the build job and with both lists of sanitisers.

Coverage

17:59 Let's run the coverage service again, and keep the rows of the server's files, the bot's and the total, with the lines each file has and the share of them that the tests ran. The world server's code went from zero to about sixty percent of its lines, its sessions to about eighty-five percent, and over all of our code, the tests run about fifty-two percent of the lines now. The bot stays at zero, as the tests have a player of their own.

18:25 In server dot c c, the broadcast's lines have counts now, from the ticks that ran while the players were in the world, while its monsters and NPCs are red, as our test's world doesn't have any. What's still red is what our players don't do: moving, talking to an NPC, chatting, getting the world over UDP, and walking through a gate.

Wrap-up

18:44 So, today the world server got a function that returns the port it listens on, so that a test can start it on port zero, with UDP on the port that TCP got. The server's tests start a world server inside the test, and play it with players over TLS, on the test's thread, waiting for each check for five seconds at most. UBSan found that our fixture destroyed its I O context before the server, which the plain build ran past. Two players see each other until one of them leaves, and an account plays once at a time. And the coverage report has the tests running about fifty-two percent of our lines, up from about forty-two, with the server's game loop among them.

Playing it

20:21 Let me know in the comments what you'd like to see next. And if you want to follow along with the series, subscribe to the channel. I hope you find this useful, and thanks for watching.