TLS for a C++ game server with OpenSSL and Asio

Intro

0:00 Hello and welcome back to episode fourteen, where we're going to make sure that what our players send over the network stays between them and our servers, starting with their passwords. Let's begin with a quick look at where we're going to be by the end of the episode.

0:15 I make an account and log in, while tcpdump shows what goes over the network to the login server. The bytes are all scrambled, so neither my name nor my password is in there, and the game works as it did before.

0:32 On the website of the series you can find more information about the project. The code for the project is open source on GitHub, tagged at each step of each episode so that you can follow along.

Last time

0:43 Now, before we get to today's work, let's quickly go over what we did last time. We started with a crash that lost the players' progress since they logged in, as the game server saved them when they left. Saving each change to MySQL fixed that, and took the server below twenty ticks a second at around fifty players. So the game server saves into Redis instead, as a hash for each player, all of them in one pipeline a tick. Redis keeps them on disk with its append-only file, and a saver of our own takes them on to MySQL, five hundred to an INSERT, once in five seconds. Measured again, the server kept up as before, and after a crash, the players came back where they were.

Today

1:23 Today, we start by looking at what our game sends over the network when you log in, with a tool called tcpdump. Then we go through what TLS is, and what it gives us, which is more than hiding the bytes. TLS needs certificates, so we're going to make a certificate authority of our own, which signs a certificate for our servers. Then we bring in OpenSSL, the library that does the cryptography, through Asio, which we've used for our networking since episode three. And we put TLS on each connection we make over TCP: to the login server, to the game server, from the client and from the bots. Last, we look at the network again, and at what all of this costs us. All right, let's get started with what goes over the network right now.

The problem

2:09 This is the game as we left it last time, with a fresh database, and with one more process on the right. Tcpdump captures the packets going through a network interface, and prints them. Dash i lo listens on the loopback, as the client and the servers are on one machine here. Dash n leaves the addresses as numbers, dash capital A prints each packet's bytes as text, and port seven seven seven eight keeps it to the login server's traffic. I type a name, alice, and a password, correct horse, and create an account. In the first packets after the connection opens, we can read my name and my password, as I typed them. The packet that comes back has the token in it, the sixty-four characters that let me into the game server for the next thirty seconds. A dot stands for a byte that can't be shown as text, such as the four bytes of each message's size. So what's left is our protobuf message, as we serialised it.

3:07 On my machine, that's harmless, so let's think about where those bytes go when someone plays over the internet. They leave our client, and go through the network the player is on, which could be a café's Wi-Fi. Then they go through the routers of each provider between there and our data centre, before they reach our servers. Anyone who can see the traffic at one of those points can read it, like tcpdump did. With the password, they can log in as you, and as a lot of people use one password in several places, it might open more than our game. With the token, they could get to the game server before your client does, and play as you. And someone in the middle can change the bytes on the way too, say, the moves our client sends. TCP's checksum doesn't stop that, as it's there to catch noise, and whoever changes the bytes can compute it again.

TLS

3:57 The fix for all three is TLS, which stands for Transport Layer Security, and is the S in HTTPS. Its older name, SSL, is still in a lot of places, OpenSSL among them. It gives us three things, and the first is encryption, so the bytes on the way are unreadable to anyone without the key. The second is integrity: each piece of data carries a tag, computed with the key, so a byte changed on the way makes the tag wrong, and the connection stops. And the third is authentication, which means our client checks that it's talking to our server, and not to someone in the middle who pretends to be it. Without that one, the first two don't help much, as someone in the middle could encrypt one connection with us, and another with the server, and read everything in between. TLS goes on top of TCP, so what we have on each side is still a stream of bytes, and our framing and our protobuf messages stay as they are.

4:52 Before any of our messages go through, the two sides shake hands, and we're going to use TLS 1.3, the newest version. The client opens with a hello, which says what it can do, and carries its half of a key exchange, a public key it made for this connection. The server answers with its own half, and from these two halves, each side works out a shared secret, without it going over the network. The method is called elliptic curve Diffie-Hellman, and OpenSSL picks a curve called X25519 for it. From here on, what the server sends is encrypted, starting with its certificate, which says who it is. Then a signature over everything the two sides have sent so far, made with the private key that goes with the certificate. Someone who copied the certificate doesn't have that key, so they can't make the signature. The client checks that it trusts the certificate, that the server's name is on it, and that the signature is right. Each side sends a finished message, and all of that took one round trip. Then our messages go through, encrypted, with AES in a mode called GCM, which gives each record the tag that makes a change show up. And as the keys come from a fresh exchange each time, someone who records the traffic today, and steals the server's key next year, still can't read it.

Certificates

6:11 A certificate is a public key with names, dates and a few rules, and a signature from whoever vouches for it. And the client trusts it because of who signed it. Whoever signs is a certificate authority, with a key that signs certificates, and a certificate of its own that anyone can check those signatures with. Our authority is going to sign one certificate, for our servers, with the names we reach them by, such as localhost on my machine. The servers present that certificate, and keep its private key, to make the signature in each handshake. And the client and the bots trust our authority's certificate, and nothing else. A website does this differently, as browsers come with a list of public authorities they trust, such as Let's Encrypt. And Let's Encrypt checks that you control a domain, and signs a certificate for it, for free. For our game, we ship the client, so it can come with our own authority in it. That works on localhost and inside Docker, where no public authority would sign anything, and on a production server too.

7:12 The keys go in a folder called tls, at the top of the repo, and private keys mustn't end up in git. So dot gitignore ignores everything in it except the script that makes them, and each copy of the repo gets an authority of its own. Dot dockerignore gets these two lines too, so a docker build doesn't copy the keys from my machine into an image. Then the script, certificates dot s h, which uses the openssl command-line tool, and starts with a shebang for a plain POSIX shell. Set dash e u stops at the first failure, like in our other scripts, so a half-made set of certificates doesn't go unnoticed. Over it, a comment says what the script does: makes a certificate authority of our own, and a certificate it signs for our servers, in the folder given, or the script's own folder by default. Any names after the folder go on the certificate too, such as a server's domain. Dir is the first argument, or the script's folder from dirname of dollar zero, and shift drops it from the arguments, so the names are what's left. We make the folder with mkdir dash p, and cd into it. If the folder has a server certificate already, we say so and stop, as a new authority would mean that clients trusting the old one stop trusting our servers. Names is the list for the server's certificate: localhost and its address, which is where we connect to our servers, and a loop adds any we were given after the folder. Then a small config file for openssl, written with a heredoc. The request section needs a distinguished name section, even an empty one, as we're going to give the names on the command line. The authority section is the rules for the authority's own certificate. Basic constraints, CA true, says it may sign other certificates, and key usage, key cert sign, says that's what its key is for. The servers section says the opposite: CA false, so a server's certificate can't sign anything. Its key makes digital signatures, which is what the handshake needs, and extended key usage, server auth, says it's for a server. Subject alt name puts our list of names on it, and that list is what the client checks the server's name against. Now the authority's key, which openssl ecparam makes with dash name prime two five six V one, on the P-256 elliptic curve, as an ECDSA key. It's thirty-two bytes, against two hundred and fifty-six for RSA, and it signs a lot faster. Umask zero seven seven makes the new file private to its owner. The parentheses run the command in a subshell, so that umask doesn't apply to the files after it. The comment over it says why it's private: the authority's key signs certificates, so nobody else gets to read it. Openssl req, dash x five oh nine, makes a certificate that the authority signs with its own key, which is how each authority starts. It lasts ten years, as each client carries this certificate, and replacing it means updating all of them. Next comes the servers' key, also an ECDSA key on P-256. Chmod six four four makes it readable to other users, as the servers read it as a user of their own, in their containers. Then a request, a C S R, which is the servers' public key and name, signed with the servers' key to show that whoever asks has the key. And openssl x five oh nine, dash req, is the authority signing it. Its serial number is sixteen random bytes, as an authority must not give two certificates one number. It lasts eight hundred and twenty-five days, which is a little over two years, and the servers section gives it its rules and names. At the end, we delete the config and the request, which we don't need any more, and say what we made. Running it again is safe, and it needs to be, as our build and our compose stack are both going to run it.

11:06 Let's run it, and openssl prints a line or two on the way, as it checks the request's own signature, and the script ends by saying what it made. In the tls folder, we now have the authority's certificate and key, the servers' certificate and key, and the script. The authority's key is readable by me and nobody else, and the servers' key by anyone on this machine. Openssl x five oh nine reads the servers' certificate back, and prints its subject, nanoMMO servers, and the authority that issued it. And it prints the names on it: localhost and the address. Openssl verify checks it against the authority's certificate and says OK, which is the check our client does in each handshake. And when we run the script again, it leaves the certificates as they are.

OpenSSL

11:54 For the cryptography, we're going to use OpenSSL, the open source toolkit for TLS, and the library behind a large part of the secure connections on the internet. Linux distributions package it, so we take it from the system's package manager, like MySQL's client library and libsodium. It has three parts, and we use all of them, starting with libssl, which is the TLS protocol, up to TLS 1.3. Libcrypto has the cryptography under it, such as the key exchange, the signatures and AES. And the openssl command-line tool is what our script used to make the certificates.

12:31 We don't call OpenSSL directly, as Asio wraps it for us, and its docs have a page on SSL. Asio's SSL stream goes on top of another stream, such as our TCP socket. And it has the reads and writes a socket has, so the rest of our code changes very little. Before that, we make a context, which holds what's set once for all the connections. On a server, that's the certificate and the key, and on a client, which certificates to trust, and whether to check them. Each connection shakes hands before it sends anything, with handshake, or with async handshake when it shouldn't block. And for the client's check that the server's name is on its certificate, Asio has host name verification, which follows the rules that HTTPS uses.

13:17 In the top-level CMakeLists, OpenSSL comes in next to MySQL's client library, which Homebrew also keeps out of the places CMake looks on a Mac. So the comment over the block from episode five now names OpenSSL and CMake too. And the block, which asks Homebrew where mysql-client lives, becomes a foreach, over it and openssl at three, the version Ubuntu twenty-four oh four has too. For each, brew dash dash prefix prints where it lives, and that goes on the CMake prefix path. Then find package OpenSSL three, required, uses CMake's own module for finding OpenSSL, and gives us an imported target for libssl, the TLS part. In the Dockerfile, the toolchain gets libssl dev, for the headers, and openssl, the command-line tool, as the build is going to run our script for the tests. In the protocol's CMakeLists, the library gets tls dot c c, as the servers, the client and the bots all link with protocol. It links with OpenSSL S S L publicly, so whatever links with protocol gets OpenSSL too. The test executable gets tls test, and test TLS names a folder in the build for that test. The comment over it reads: the TLS test's certificates, from two authorities, ours and someone else's. A foreach makes the two authorities in it, each with add custom command, which runs our script with the folder as its argument. Output says what file the command makes, and depends says what it's made from, so the build runs the script when the certificates are missing. A custom target depends on both, and the tests depend on it, so the certificates are there before the tests are built. And target compile definitions gives the tests the folder, as a macro, so they can find the certificates wherever the build is.

15:07 Now the two contexts, in a new header in protocol, tls dot h, with its include guard. It starts with string, for the paths to the files, and Abseil's status or. Then three headers from Asio: TCP, its SSL context, and its SSL stream. TlsStream is a short name for Asio's SSL stream on top of a TCP socket, as we're going to write that type in a few places. Server TLS returns a status or of an Asio SSL context, so a missing or broken certificate or key comes back as an error that main can print. Its comment spells it out: TLS 1.3 for a server, with its certificate and its private key, from PEM files, which is the text format our script wrote them in. Client TLS makes the context for a client, and its comment says what that means: TLS 1.3 for a client, which trusts the certificates our authority signed, and no others. It takes the path to the authority's certificate, as that's all a client needs. In tls dot c c, after our header, string, and system error, for the error codes Asio gives back. Then Abseil's status and str cat, and Asio's verify mode, while status or and the SSL context come with our header. In an unnamed namespace, a helper, as both functions load files, and their errors should read alike. Load error is for a file that didn't load, and it takes the file's name, and what Asio said about it. It returns a failed precondition error, which is Abseil's code for something that has to be in place first. Next to Asio's reason, the message says that our script makes them, as that's the fix for whoever sees it. Then the two functions from the header, and server TLS comes first, with the signature it has there. It makes a context for TLS 1.3 as a server, and nothing older. We write both ends, so there's no old client to support, and TLS 1.3 dropped the older ciphers and options. Use certificate chain file loads the certificate, along with any authorities between it and the top, which we don't have. As we don't use exceptions, we take the overload with an error code, and return load error if it's set. Use private key file loads the key, in PEM, and OpenSSL checks that it goes with the certificate. So a key from another set of certificates fails here, at the start. That error goes through load error too, and the return moves the context into the status or, as a context can't be copied. Client TLS makes a context for TLS 1.3 as a client. Load verify file makes it trust the certificates in the authority's file. And we don't load the system's list of public authorities, so a certificate from one of those doesn't get in either. Set verify mode, verify peer, makes the client check the server's certificate in each handshake. OpenSSL's default for a client is not to check, and then it would encrypt our traffic for whoever answered. So these lines make our authority the one our clients trust, by loading its certificate and checking the server's against it.

18:15 The test goes in tls test dot c c, and it runs both ends of a handshake in one process. After our header come string and system error, then Abseil's status or and str cat. From Asio, error, io context, address, TCP, host name verification and stream base, and then gtest. In an unnamed namespace, we start with two constants for the test's folders. K ours is our authority's folder, from the macro the build gave us, with a slash on the end. Over the two goes a comment: two authorities the build made for this test, ours, and someone else's. And k theirs holds the folder of the other one. Connects is the helper that shakes hands, and it takes the server's context, the client's, and the name the client expects, and returns whether the handshake worked. It has an io context of its own, and an acceptor on one two seven dot oh dot oh dot one, port zero, which asks the system for any free port. Then a TLS stream for each side, both on that io context. The client's verify callback is host name verification with the name we passed in, so the name is part of the check. The acceptor takes the server's side of the connection, and the server shakes hands, as a server. Its handler ignores how that went, as the client's side tells us what we want to know. Result starts as not connected, so if the client doesn't get as far as a handshake, the helper returns false. The client connects to the acceptor's port, and shakes hands as a client, and result keeps how that went. These calls just queue up work, and io run does it, both handshakes at once, on this one thread, until there's nothing left to do. Then the helper returns true if result holds no error, and we go back up to write its comment. A client that checks the certificate for host's name shakes hands with the server, over this machine's loopback. The test is called trusts our authority, and it's one test, as these checks belong together. First, the server's context, from the server certificate and key in our folder, joined with str cat, so this is our server as the game would run it. Then a client context that trusts our authority, and one that trusts theirs, from the other folder. Assert true checks that all three loaded, and stops the test there if they didn't, as the four checks after it would mean nothing. A client that trusts our authority, and expects localhost, connects. A client that trusts theirs doesn't, and that's someone in the middle, with a certificate of their own. A client that expects nanommo dot example dot com doesn't either, as that name isn't on the certificate. And client TLS with a file that isn't there returns an error.

20:58 After a build, the build folder has the test's two authorities, ours and theirs, each with its own certificate and key, and a certificate and a key for its servers. And the TLS test passes, which means a client that trusts our authority shakes hands with our server, and the handshake fails with their authority, or with another name.

Both ends

21:18 Now the servers, and both of them talk to their clients through Session, from episode three, so that's where the encryption goes in. In session dot h, the header we need from Asio, and ours from protocol. The constructor takes the context too, after the socket, by reference, as all the sessions share one. Start's comment says that it shakes hands over TLS, then reads the client's messages, and further down, the socket member becomes our TLS stream. In session dot c c, stream base, from Asio's SSL folder, which names the two sides of a handshake. The constructor takes the context after the socket, like the header says, which pushes the functions for messages and for closing onto a second line. The initialiser list makes the stream from the socket and the context. And the stream takes the socket over, and makes its own state for this connection from the context. Turning off Nagle's algorithm is an option of the TCP socket under the TLS stream, which we reach through lowest layer. Start used to read straight away, and now it starts the handshake first, as the server. Async handshake doesn't block, so the server keeps ticking while a player connects. Its handler keeps the session alive, like our reads do. If the handshake fails, say for a client that doesn't trust us or doesn't speak TLS, we close the session, and otherwise we start reading. The read and the write go through the stream now, and the write still hands over three buffers: the size, the world and the player's own part. Asio's TLS stream copies buffers together, up to eight kilobytes, before it encrypts them, so the four bytes of the size don't go out in a record of their own. And close closes the TCP socket under it, through the lowest layer. TLS has a message for closing a connection cleanly, and we don't wait for it. And that's fine, as our messages carry their sizes, so a cut-off connection can't pass for a shorter message.

23:13 The login server needs a context to hand to its sessions, so we start in its header, login server dot h, with the header we need from Asio. Create's comment gets a second sentence: each client talks to it over TLS, with the context. Create takes the context after the port, by reference, as main is going to make it and keep it alive while the server runs. The private constructor takes it right after the io context, and the class keeps the reference in tls underscore, next to the acceptor, to hand to each session it makes. In login server dot c c, create's definition passes the context on to the constructor, through wrap unique, like the other arguments. The constructor keeps it next to the acceptor, in the order the class declares its members, as we saw in episode four. Further down, accept makes a session for each client, and the session now needs the context, along with the functions for its messages and for when it closes. The login server has nothing to do when a client goes, so on close is an empty lambda. And it now has a name, so that the line making the session fits in eighty columns. Then make shared makes the session with all three, and start begins with the handshake. The game server gets these changes too, starting in server dot h with the header for the context. Create's comment gets a sentence too: clients connect over TLS, with the context. Like the port, the context is about how players connect, so create takes it right after the port. The interval, the metrics, the tokens and the players come after it, as they're about what the server does once players are in. The private constructor takes it after the io context as well, and clang-format wraps both lists of parameters. And tls underscore holds the reference, under the acceptor, where the connections come in. In server dot c c, the SSL context comes with our header, so the changes start further down. There, create's definition gets the context between the port and the interval, like the header, and passes it on to the constructor, through wrap unique. The constructor takes the context and keeps the reference in tls underscore, right after the acceptor, like in the login server. Then accept, where each session gets it too, calls start on what make shared returns, without the variable we had. The session lives on through the handlers it starts, which hold a shared pointer to it. The client and the bots are going to need settings too, for where our authority's certificate is. They don't link with the server's code, so settings dot h moves from server to protocol, which all of them link with. Setting is our function from episode six, which reads an environment variable or falls back to a default, and its header guard follows the new path. In the game server's main dot c c, the header from Asio, then settings from its new place, and our tls dot h. After the port, and before anything else is set up, server TLS loads the certificate and the key. Nanommo TLS certificate and nanommo TLS key say where they are, with the tls folder our script fills as the default, and in Docker, compose points them somewhere else. If loading them fails, we print why, with the hint to run the script, and return one. Stopping at the start with a clear message is better than starting and having each player's handshake fail. Then server create gets the context after the port, and as main declares tls before the server, tls is destroyed after it, so the reference in the server stays valid. The login server's main gets those three headers too: the one from Asio, settings, and our tls dot h. At its very start, before it connects to MySQL and Redis, it loads the certificate and the key, so a missing certificate stops it right away. If that fails, it prints nanommo login in front of the error, and returns one. Further down, login server create gets the context, after the port, like the game server's. And the saver needs nothing more than the include from its new place, as it doesn't talk to any clients.

27:12 On the other end, the client and the bots go through log in, the function in protocol from episode six. In login client dot h, one header from Asio, for the type. The comment on log in now says that the request goes over TLS, and log in takes the context after the port. In login client dot c c, two headers from Asio's SSL folder, host name verification and stream base, and our tls dot h. Log in makes an io context of its own, which lives for the time the call takes. The socket becomes a TLS stream, on that io context and the TLS context we were given, and connect goes through the stream's lowest layer. Before the handshake, the verify callback: host name verification with the host we connected to, so the certificate has to have that name on it. Then handshake, as a client, which blocks until it's done, and that's fine, as log in blocks anyway until the login server answers. If any of these steps fails, the ones after it don't run, and we return its error at the end. Writing the request and reading the reply go through the stream now. The game client's connection to the game server gets TLS like this too, and it starts in connection dot h, with the header from Asio and our tls dot h. Create's comment now says that it connects over TLS, and create takes the context after the port, which pushes the token and whether to use UDP onto a second line. The constructor takes the context along with the token, so it isn't explicit any more, as with two arguments it can't work as a conversion. Last in the header, the socket member becomes a TLS stream, stream underscore. In connection dot c c, two headers from Asio's SSL folder: host name verification and stream base. Create's definition takes the context, like the header says, and passes it to the constructor, along with the token. The constructor makes the stream on the connection's io context, with the TLS context. Then start, which connects to the game server, and says hello with the token. The resolver finds the server's addresses, as before, and connect goes through the stream's lowest layer, the TCP socket under it. And turning off Nagle's algorithm goes through it too, like in the session. Then the verify callback with the host name, like in log in, and a handshake that blocks like connecting does, as the login screen waits for it. If it fails, we return an unavailable error with OpenSSL's reason, so the login screen can show it. A certificate we don't trust shows up there as certificate verify failed. Then the hello goes through the stream, so the token is encrypted now too. And for UDP, the server's address comes from the lowest layer, as the TCP socket under the stream has it. Sending input goes through the stream, and blocks as before, while a read waits for the next world. A write in TLS 1.3 doesn't need anything from the server, so it's fine next to a read that's waiting. And the read, with async read some, goes through the stream too. In the client's main dot c c, the header from Asio, our settings and our tls dot h. Enter takes the context, and hands it to log in and to the connection. And at the start of main, before the window opens, client TLS loads our authority's certificate, from nanommo TLS C A, or from the tls folder by default. If it isn't there, we print why and stop, and last, the call to enter passes the context.

30:42 The bots get encrypted connections too, so that our load tests pay for them like real players would. In bot dot h, the header from Asio, and ours from protocol, for the stream's type. The constructor takes the context after the io context, as each bot makes its stream from it. Then a new private function, greet, for what a bot does once it has shaken hands, which takes the token and the handshake's error code. And the socket member becomes a TLS stream, like the client's. In bot dot c c, two headers from Asio's SSL folder, like the client's: host name verification and stream base. The constructor takes the context, and makes the stream on the io context with it. A thousand bots share one thread, so a bot mustn't block, and everything it does is a chain of handlers, each one starting the next step. Start used to connect, and say hello in the connect's handler. Now it starts with the verify callback, and its comment says where the name comes from: the resolver keeps the name we looked up, which the certificate must have. So host name verification gets the host name from the first result. Then async connect connects the stream's lowest layer, with the handler it had. In that handler, an error still prints why and returns, and Nagle's algorithm goes off on the TCP socket under the stream. Then async handshake starts the handshake, without blocking, and its handler calls greet, with the token and the handshake's error. Greet is what used to come after connecting, moved into a function of its own, so the handshake's handler stays short. If the handshake failed, it prints why, and that bot stops there. Otherwise, it says hello with the token, with its comment from before: blocking is fine, as a hello this small fits in the socket's buffer. With TLS, the hello is encrypted first, and it's still under a hundred bytes. If the write failed, we stop there too, and otherwise the bot counts as connected, reads, and starts wandering. And with UDP, it finds the server's address through the lowest layer, connects its UDP socket, keeps the token, and says hello over UDP, as before. Further down, read and wander go through the stream too. In the bots' main dot c c, the header from Asio, settings and tls dot h. Our log in helper takes a context, and passes it on, for both tries: logging in, and making the account when the bot is new. The swarm takes the context too, keeps a reference to it, and hands it to each bot it makes, and to each login. Main makes the context once, from nanommo TLS C A like the client, and all the bots share it. So a thousand bots don't load the file a thousand times. If the authority's certificate isn't there, we print why and stop, and last, the swarm gets the context, as its second argument.

33:32 Last, Docker, where the runtime stage, which the servers' images start from, gets OpenSSL's shared library, libssl three T sixty-four. The bots' image starts from plain Ubuntu, so it installs the library too, instead of counting on the base image having it. Then a small stage, certificates, with Ubuntu and the openssl tool, and our script on the path as its entry point, so running the image runs it. In compose dot yaml, that becomes the certificates service, which writes to slash tls, where our tls folder is mounted, so the certificates land in the repo. So the first docker compose up makes them, and after that, the script leaves them as they are. The login server gets nanommo TLS certificate and key, pointing into slash tls, and the folder, mounted with r o, so the server can read the certificates but not change them. And it depends on certificates with the condition service completed successfully, so it starts once the script has finished. Compose's plain depends on waits for a service to start, and the script would still be writing then. The game server, further down, gets those settings, the folder, and the wait for the certificates too. The bots get nanommo TLS C A and the folder, mounted with r o too, for the authority's certificate. And CI builds the certificates image, like our other images, so a mistake in it shows up there.

34:58 Let's start the login server in Docker, from a fresh copy of the repo. Docker compose up builds the images, and runs the certificates service next to MySQL and Redis. Then it starts the login server, once the certificates are made and the other two are healthy. The tls folder now has the certificates, which the script made in its container. To check the login server from the outside, openssl has a client, s client, which connects and shakes hands, and dash brief prints a summary. We give it our authority's certificate with dash C A file. The connection is established with TLS 1.3, and AES with a two hundred and fifty-six bit key, in GCM. The server's certificate is nanoMMO servers, the server signed the handshake with ECDSA on P-256, and verification is OK. The temporary key, X25519, is the key exchange from the handshake. Without our authority's certificate, openssl uses the system's list of public authorities, and none of those signed ours. So it can't find the certificate's issuer, the verification fails, and with verify return error, the handshake stops there. And with our authority, but expecting another name, the check fails with hostname mismatch, as it would in our client's host name verification.

36:18 Now the game server, in Docker too, along with the certificates, MySQL and Redis it depends on. And five bots, over five seconds, with bots and ramp, which compose dot yaml reads from the environment. Compose starts the login server too, as the bots depend on it. Each bot says it couldn't reach the login server, as the certificate verify failed. The bots connect to login and server, the services' names on Compose's network, and host name verification checks the name they connect to against the certificate. Our certificate names localhost and its address, and nothing else, so for the bots, it's a certificate for another host. We made that mistake with Prometheus in episode four, as inside Compose, our servers aren't on localhost.

37:03 So we go back to the script, to the line where names lists localhost and its address. We add login and server, the names Compose gives our two servers on its network, so that the bots, and anything else inside Compose, find them on the certificate. Both servers share this one certificate, and its key, so the certificate names both of them.

37:24 The script leaves certificates that are there already as they are, so we delete the ones it made, the certificates and their keys. Docker compose up then runs the certificates service again, which makes new ones, before the game server starts. The servers' certificate now has login and server on it, next to localhost and the address. Let's start the five bots again, and this time, they log in, and join the game server, over TLS.

Looking again

37:49 Now let's repeat the analysis with TLS in place, with tcpdump on the loopback again, for the login server's port. I type the name, alice, and the password, correct horse, and create the account. The first packets are the handshake, with the two hellos, and from the server's hello on, the rest is encrypted, the certificate along with it. Our request comes next, as a short run of random-looking bytes with no alice and no correct horse in it. And the reply with the token looks like that too. And the game works as before, with the token and our moves going to the game server over TLS too.

The cost

38:31 To see what all of this costs, we run openssl speed, which measures how fast OpenSSL does each of these steps. It runs in our certificates image, which has Ubuntu's OpenSSL, like our servers. First, the signature the server makes in each handshake: with ECDSA on P-256, my machine makes around ninety thousand a second, about eleven microseconds each. RSA with a two-thousand-and-forty-eight-bit key, which a lot of servers still use, makes about three thousand, about thirty-two times slower. So our script makes ECDSA keys, as the game server shakes hands on the thread that runs the tick. The key exchange, X25519, is around fifty thousand a second, so twenty microseconds. Together, a handshake costs the server well under a tenth of a millisecond, plus a round trip. Logging in already takes tens of milliseconds, for Argon2, as we measured in episode five, so the handshake doesn't change much there. Then the encryption of our messages, AES-256 in GCM, which gets about seven point three gigabytes a second for records of a kilobyte, and about eight point one for sixteen kilobytes. The processor has instructions for AES, so the hardware does most of the encryption. The worlds go over UDP by default, and over TCP it's the inputs and the hellos, so for the game server, that's a few microseconds a tick. With the world over TCP, the server encrypts each player's copy of it, and for a thousand players, that's a few milliseconds a tick.

What stays plain

40:06 Let's go through what TLS covers in our game, and what it doesn't. Everything over TCP goes through it: logging in, the token, the game server's hello, and the moves our client sends. The world has gone over UDP since episode nine, and TLS needs TCP's stream, so it doesn't cover that. The world is where each player is, and their names and levels, which anyone logged into the game sees anyway, so reading it doesn't give away much. The UDP hello carries the token, so the game server can look up which player that address belongs to. By then, the game server has taken the token out of Redis, so it can't log anyone in. Someone who can see our traffic could send a hello of their own with it, and have the game server send your world updates to them. Someone in that position could also drop the updates on the way, so it doesn't give them much more than they had. If we wanted UDP covered too, there's DTLS, which is TLS for datagrams, or encrypting each datagram in our own code, with a key we agree over the TLS connection. Either one is more work on each datagram, and for what we send over UDP today, we're going to leave it plain.

Docs

41:15 Last, the README, where the list of things to install gets OpenSSL, with Homebrew's package and Ubuntu's, and running the game locally runs our script before the login server. A new section on TLS says how to put your own domain on the certificate, with names after the folder, and how to make new certificates, by deleting the old files. And it says what goes where on a production server: the servers need their certificate and key, and the players' clients the authority's certificate. The authority's key stays somewhere else, as it can sign certificates our clients trust. It ends with the s client command, for you to look at the handshake.

Wrap-up

41:53 So, today we saw our login go over the network with tcpdump, with the password as I typed it, and the token on its way back. TLS gives us encryption, integrity, and a server that proves who it is, in a one-round-trip handshake in TLS 1.3. We made a certificate authority of our own, with ECDSA keys on P-256. And it signed a certificate for our servers, with the names we reach them by. Two functions make OpenSSL's contexts, through Asio: one for the servers, with their certificate and key, and one for the clients. And the one for the clients trusts our authority, and checks the server's certificate against it. Then each TCP connection went over TLS 1.3: the login server, the game server, the client and the bots, with Docker Compose making the certificates on its first run. A test, and openssl s client, showed that a certificate from another authority, or a name that isn't on ours, gets refused. Tcpdump again showed nothing of our messages that we could read. A handshake costs the server well under a tenth of a millisecond, and encrypting runs at gigabytes a second. And the world over UDP stays plain, as it holds what players see in the game anyway.

Playing it

44:12 Let me know in the comments what you'd like to see next. And if you want to follow along with the series, subscribe to the channel. I hope you find this useful, and thanks for watching.