Self-hosted GitHub Actions runner: deploy on push

Intro

0:00 Hello and welcome back to episode thirty-seven, where our server is going to deploy each push to main on its own, once CI passes. Let's begin with a quick look at where we're going to be by the end of the episode.

0:13 I push a commit to main, which starts a run of CI on GitHub. G H run watch follows it, sped up: CI checks the commit on GitHub's runners, and once it passes, the deploy job runs on a runner on our server, and deploys it.

0:29 On the website of the series you can find more information about the project. The code for the project is open source on GitHub, tagged at each step of each episode so that you can follow along.

Last time

0:41 Now, before we get to today's work, let's quickly go over what we did last time. Each instance of a world registered with Consul under a name of its own, and the login server lists the newest. On the signal to stop, a world saves its players and sends each client a move to its next instance, with a fresh token for each move. The client moves over with the world still on its screen. Handover dot sh restarts a world through its next instance, and the deploy script restarts the worlds that way, and the bots follow a move like a client does.

Today

1:14 Since episode nineteen, a deploy has been deploy dot sh, run from my machine, which sends a commit to the VPS over SSH and starts it there. And since episode one, GitHub Actions checks each push to main, with our tests, and since episode twenty-eight, with the sanitisers and the fuzzers too. So we're going to put the two together, and let a push that passes CI deploy on its own. A GitHub Actions runner is the program that runs a workflow's jobs, and GitHub lets you run your own, self-hosted, so we're going to run one on our server, in Docker. Once CI passes for a push to main, the runner deploys it, through the worlds' next instances, so the players play on. And as our repository is public, the runner is going to refuse any job but that deploy, as it can do anything on the server. All right, let's get started, with the half of the deploy script that runs on the server.

A release, from the server

2:09 Release dot sh is a new script at the top of the repo, and it starts a release on the server. Until now, most of that was the script deploy dot sh ran over SSH, so we're moving it into a file each release carries, where something on the server can run it too. Set dash e u stops it at the first command that fails, like our other scripts, and above it we write what the script does. It starts the release in the folder this script is in, nanommo dot next on a server, such as a commit deploy dot sh sent. It builds the release's images, and if they build and the tests pass, the release becomes nanommo, the one that runs. Dirname of dollar zero is the folder the script is in, nanommo dot next, whichever folder it's called from, and dot dot takes us up to the folder with the releases. Over SSH, that was the user's home folder, and now it's wherever the releases are. If a release runs there already, in nanommo, the new release gets its dot env, with the database's password and the website's key, as deploy dot sh did. Then, if the new release has no server key in its T L S folder, we copy the authority's certificate, and the server's certificate and key, from the release before. A release without certificates keeps the ones the release before has. Deploy dot sh sends them with each commit, from my machine, where the authority's key stays, so a release that comes from the server has none of its own. Without a release there, this is the server's first, which needs the address players reach the server at, for the worlds to hand out to the clients. The colon is a command that does nothing, and the question mark after the variable's name stops the script, with the message after it, when public host isn't set. So without an address, the script stops with that message before it makes anything. Above it, we write that the first release on a server needs the server's address, from deploy dot sh. The rest of the branch is what deploy dot sh did on a first deploy. Dot env gets Compose's files, a random password for MySQL, and a random key for the website, and nobody but our user can read the password and the key. A dot env without the public host gets it, as before. Then the images build in nanommo dot next, the next instances' too, and the tests run as part of the build. A build that fails, or a test, leaves the release that runs as it is, as set dash e u stops the script there. Once it's built, the release before moves into nanommo dot previous, in place of the one before that, and the new release becomes nanommo. In there, Compose starts each container again from the new release's folder, apart from the worlds, which grep leaves out of the list of services. The worlds start again through handover dot sh, Ember first, and then Frost. The reason goes above the up command: a container left running would keep the folder it started from, which is nanommo dot previous now. The worlds start again through their next instances, one at a time, so that their players play on. In deploy dot sh, the header changes to say that it starts the commit there with release dot sh. The script it sends over SSH still unpacks the commit into nanommo dot next, and everything after that goes, as release dot sh does it now. In its place, it runs release dot sh from the release it just unpacked, with the server's address in public host. The first line of the script it sends sets the address as a plain shell variable, which a child process such as release dot sh doesn't see, so it goes in front of the command. Release dot sh joins deploy dot sh in dot docker ignore, like the compose files in episode four, so a change to the deploy scripts doesn't build the game again. And in the README's section on deployment, the paragraph on what the server does with a commit now says that release dot sh starts it. A new paragraph says that release dot sh starts any release unpacked into nanommo dot next, next to nanommo, so that something on the server can deploy too. A release without certificates keeps the running release's, and the first release on a server comes from deploy dot sh, which passes the server's address.

6:26 For this episode, a Debian virtual machine on my laptop stands in for our VPS, with Docker and the firewall set up like the one from episode nineteen. Let's deploy this commit from my machine, as before, to check that deploy dot sh still works with release dot sh doing the work on the server. The server builds the images and runs the tests, which takes a while, so this is sped up. On the server, nanommo is the new release, and nanommo dot previous the one before.

A runner on the server

6:56 GitHub queues each workflow's jobs until a runner takes them. A runner connects to GitHub and asks for work, so our firewall needs no new port open for it. Each runner has labels, such as self-hosted, and a job says which labels it runs on, so our deploy job is going to ask for a runner labelled nanommo. The runner runs in a container, and a deploy runs Docker Compose, so the container gets Docker's socket, which is how a Docker client talks to the server's Docker. And it goes in a Compose project of its own, as a deploy starts each container of the game's project again, and the runner's container is the one running the deploy.

7:36 The runner gets a folder of its own, runner, with a Dockerfile for its image. We start from GitHub's own runner image, from GitHub's container registry, at the runner's version two point three three eight. It comes with the runner, Docker's command line client and git, and it runs the runner as a user called runner. Over it, the header reads: a self-hosted GitHub Actions runner for the server, from GitHub's runner image. It adds Docker Compose, which a deploy runs on the server's Docker. Compose isn't in GitHub's image, so we copy it in from compose bin, an image that Docker publishes with nothing in it but Compose's binary. Copy dash dash from takes the binary out of that image, like we did with Composer's for the website. Compose goes into the folder where Docker's client looks for its plugins, so that docker compose works like it does on the server. GitHub's image sets a variable that prints the runner's diagnostic log to its output, which is several lines for each thing the runner does, so we set it to nothing. As the comment over it reads, the runner says what it does, without its diagnostic log, which is what we want to read in Docker's logs. And our entrypoint script goes into the image, in a folder on the path, as what the container runs when it starts. Entrypoint dot sh is what the container runs, and a runner has to be registered with a repository before GitHub sends it any jobs. A runner registers once, with a token from GitHub that lasts an hour, and gets credentials of its own back, which it keeps from then on. Set dash e u goes first, like in our other scripts, with a header above it. It registers the runner with the repository at runner URL the first time its container starts, with a registration token in runner token, and then waits for jobs. GitHub's image has the runner in slash home slash runner, so we go there first. Config dot sh, the runner's own script for its settings, writes the registration into a file called dot runner, so a container without that file hasn't registered yet. Unattended means it asks nothing and takes what we pass it, and replace takes the place of a runner with that name that's registered already, such as the one in a container we've replaced. The runner's name is nanommo, and labels adds the label nanommo to the ones it gets on its own: self-hosted, Linux, and the machine's architecture. Our deploy job is going to ask for those labels. The repository's URL and the token come from the environment, and each stops the script with a message when it's missing, with a colon and a question mark, as in release dot sh. Then the script starts the runner with exec, which replaces the shell with run dot sh. So the runner is the container's first process, and it's the runner that gets Docker's signal to stop. The runner's compose dot yaml is a Compose project of its own, called nanommo runner, with the reason written over its name. As we saw in the diagram a moment ago, a runner in the game's project would be stopped by the deploy it's running, in the middle of its own job. The runner's service builds its image from the Dockerfile next to it. It passes the repository's URL, and the token, from the environment Compose runs in. The dash after the colon makes each of them empty when it isn't set, as a runner that's registered needs neither. Then we mount Docker's socket, the file the Docker daemon listens on, into the container, where Docker's client looks for it. That's the server's Docker, which a deploy builds and starts the game with, so the game's containers run next to the runner's, and not inside it. On the server, the socket belongs to root and to the docker group, and the runner's user in the container isn't in that group, so connecting to the socket would fail with permission denied. Group add adds a group to the user the container runs as, on top of the groups it has, by the group's number, from docker group. A group's number can differ from one machine to the next, and GitHub's image has a docker group of its own, numbered a hundred and twenty-three. Above group add, we write that the socket belongs to the server's docker group, which the container knows by its number alone. And restart unless stopped starts the runner again after a crash or a reboot, like our services in production. Start dot sh starts the runner on the server. With set dash e u again, the header says how to use the script. It starts the runner on this server, for a GitHub repository, such as ours, with a registration token, which a new runner needs. A registration token comes from GitHub's API, through G H, which is GitHub's command line tool, and making one takes admin rights on the repository. The script goes to its own folder, which is the runner folder of the release that runs, and hands Compose the repository's URL and the token, from its first and second arguments, in its environment. Stat dash c percent g prints the group of the socket, as a number, for group add, and Compose builds the image and starts the container, in the background. The game's builds leave the runner's folder out, like release dot sh, so that a change to the runner doesn't build the game again. And in CI, the build job builds the runner's image too, after the servers' images, so that a change that breaks it shows up on GitHub, before it reaches the server. And the README gets a section on deploying on push, with what the runner is, how to start it on a server, and where its token comes from.

12:59 First, we ask GitHub's API for a registration token, through G H. It lets a runner register with our repository for an hour. I've deployed this commit with deploy dot sh, so start dot sh is on the server, and with the repository and the token, it builds the runner's image there and starts its container. The runner's log says it connected to GitHub, and it's listening for jobs. And GitHub lists it, online, with its labels.

A deploy job

13:31 In CI's workflow, a new job, deploy, goes in after the fuzzers and before the docs. Its if says when it runs, over the next two lines, as the greater than sign folds them into one. It's for a push to main, as a pull request or another branch isn't something we deploy, and in a repository whose nanommo deploy variable is true. Vars are a repository's configuration variables, which we set in its settings or with G H, and which aren't secret. A job waits for a runner with its labels for up to a day, and then GitHub cancels it. So a fork of our repository without a runner of its own would have a deploy job waiting after each push to main, and the variable leaves the job out there. Above the if, we write that a repository with a runner on its server says so in its nanommo deploy variable, and elsewhere, the job would wait for a runner that doesn't come. Needs makes it wait for the build, the two sanitiser jobs and the fuzzers, so that it starts once they've all passed, and a commit that fails any of them isn't deployed. Runs on asks for a runner with the labels self-hosted and nanommo, which is ours, where our other jobs ask for ubuntu latest, one of GitHub's. Concurrency puts each deploy in a group called deploy, and GitHub runs one job of a group at a time, across all the repository's runs. Above it, the comment reads one deploy at a time, and a push while one runs waits for it, as cancel in progress is off by default. And a newer push takes the place of one that's waiting, so the newest commit is deployed, and no deploy is stopped halfway, with the worlds on two releases. The job's steps check the commit out first, into the runner's workspace, like our other jobs. Then the deploy, as a shell script, which goes to the home folder, where the releases are. It removes what's left of a release that didn't start, and makes nanommo dot next again. Git archive writes the commit we checked out as a tar, from the workspace, which GitHub workspace points at, and tar unpacks it in nanommo dot next. The tar is what deploy dot sh sends over SSH, without the certificates, as the release keeps the ones it has. And release dot sh starts it, from the release we've just unpacked. And the README says how to switch the job on, with G H variable set, and how the deploys wait for each other.

15:57 We switch the deploy job on, with the variable, and push to main, which starts a run of CI on GitHub. G H run watch follows it, sped up, as it takes a while. The build, the sanitisers and the fuzzers pass on GitHub's runners, and then the deploy job runs on ours, and fails. In the deploy job's log, grep finds release dot sh's message, the first release on a server comes from deploy dot sh. So release dot sh found no release where it ran, and the job ran it in the runner's home folder, which is slash home slash runner, in the runner's container. The server's home folder, with the releases, isn't in the container at all.

The server's home folder

16:38 So in the runner's compose dot yaml, the runner is going to work in the home folder of the user on the server who starts it. Compose fills in dollar home from the shell it runs in, on the server, and the runner gets it as nanommo home, which the deploy job is going to read. Inside the container, the runner's own home stays slash home slash runner. Without that line, a user who isn't in the container's list of users gets slash as their home, where Docker and git can't write their settings. The comment above it reads that the container may have no home for the user, so they get the runner's folder, where Docker and git keep their settings. User sets who the runner runs as, by user and group ids, which start dot sh is going to pass. Above it, we write that it runs as the user who started it, so that it can write in their home folder, and what it writes there is theirs. A release the runner deploys then belongs to the deploying user, like one from deploy dot sh, so a deploy over SSH can still replace it, and the runner can replace one that deploy dot sh made. The comment about the server's Docker moves down, over the socket, as volumes is going to have two entries. The second is the home folder, mounted into the container at the path it has on the server. The path has to match, because the runner's Compose talks to the server's Docker, which starts the game's containers on the server, outside the runner. Say, Flyway's dot slash d b becomes slash home slash debian slash nanommo slash d b, worked out from the folder Compose runs in. As the comment over the mount says, Compose hands Docker the paths of what a container mounts, and Docker looks for them on the server. So the runner has to see the releases where the server has them. In start dot sh, the runner's user goes in next to the URL and the token. Nanommo user is the user's id and group, from id dash u and id dash g, with a colon between them, which is how Compose's user takes them. Id dash u prints the user's number, such as one thousand for the first user on a Debian server, and id dash g the number of their group. The line was getting long, so the docker group goes on the next line, with Compose. And the deploy job goes to nanommo home, the home folder of the user on the server, in place of the runner's own. And the README says where the runner works, and why its path in the container matches the server's.

19:10 The runner's compose file changed, so I've deployed this commit by hand and started the runner again, with a new token. With forty bots playing on the server, let's push it to main, and watch the run. CI passes on GitHub's runners, sped up again, and the deploy job builds the release on the server, starts it, and hands each world's players over to its next instance and back. In the bots' log, grep finds the lines about moving, which say that all forty moved, to the next instance and back. And the last count of connected bots that the swarm logged is forty of forty.

Who gets the runner

19:47 The runner can start any container on the server through Docker's socket, with any of the server's folders mounted, which is as good as being root there. Our repository is public, and a pull request from a fork runs the workflow as the pull request has it, so it could add a job of its own with our runner's labels, and run anything on our server. For that reason, GitHub recommends keeping self-hosted runners to private repositories. So the runner is going to check each job before it runs it, from a script in its image, which a pull request can't change. And the repository's settings are going to have workflows from forks wait for our approval too.

20:23 Job started dot sh is new, and it's going to be the runner's job started hook. A runner can run a script of ours before each job it takes, and when the script exits with an error, the runner fails the job. Its if refuses anything but a push to main, and we write the reason at the top. It runs before each job the runner takes, from the runner's image, which no pull request can change. The runner has the server's Docker, so it takes deploys of pushes to main and nothing more, and a job it refuses fails before its first step. The variables in the if are ones GitHub gives each job. GitHub event name says what started the job, such as a push or a pull request. GitHub ref is the full name of what the job is for, refs heads main for our main branch, and one under refs pull for a pull request. We write what we refused into the job's log, with the event and the ref, and why, on standard error, where an error belongs. Then we exit with one, for the runner to fail the job. In the runner's Dockerfile, the script goes into the image, next to the entrypoint, under a comment that the runner checks each job before it runs it. Actions runner hook job started points the runner at the script. The runner reads that variable when it starts, and runs the script before each job. So the check comes with the image, and the runner on the server runs it once start dot sh has started the runner again, with a new token. Someone who can push to main can change the game, and the deploy, anyway, so a push to main is the job the runner takes. And the README says why the runner refuses other jobs, and to have workflows from forks wait for our approval, in the repository's settings, with the option's name as GitHub has it.

22:13 Let's try the script with what a pull request's job would have, and it refuses it, with its message. With a push to main, it exits with zero, so echo prints taken, and on the runner the job would run.

Wrap-up

22:26 So, today the half of the deploy that runs on the server moved into release dot sh, so that the server can start a release too. GitHub's runner runs on our server, in Docker, with Docker's socket, in a Compose project of its own. A deploy job runs on it for each push to main, once CI passes, one deploy at a time. The runner works in the server's home folder, at the path it has on the server, as the user who started it, and forty bots played on through a deploy. And the runner refuses any job that isn't a deploy of main, as a pull request could otherwise run anything on our server.

Playing it

23:51 Let me know in the comments what you'd like to see next. And if you want to follow along with the series, subscribe to the channel. I hope you find this useful, and thanks for watching.