MMO items in C++ with protobuf text format

Intro

0:00 Hello and welcome back to episode twenty-six, where we're going to give our game an item system, starting with the sword our players hold, kept in a single file that both the server and the client read. Let's begin with a quick look at where we're going to be by the end of the episode.

0:16 As the camera turns around alice, she holds the sword as before, while its model, its length and where her hand grips it come from an items file that the client reads. Her swing takes its reach, its timing and its damage from that file too, on the server, which reads the file as it starts.

0:33 On the website of the series you can find more information about the project. The code for the project is open source on GitHub, tagged at each step of each episode so that you can follow along.

Last time

0:45 Now, before we get to today's work, let's quickly go over what we did last time. One of our NPCs became a storyteller, which writes you a short story from a few words with mvGPT, the small language model from my other series, whose server runs in Compose next to the game. Your words travel with your input, and the server checks how close you are, cleans the words and limits how often you ask. Then it asks for the story from sixteen threads of its own, so that the tick doesn't wait for the model. On the client, a ray from the mouse picks the storyteller, and an ImGui window takes your words. The story comes back as lines of chat, after a first try where it was too long for a line, and went nowhere. Last, the dashboard counted the stories, and our bots asked for them too.

Today

1:30 Today, our sword lives in two places, as the server has its reach and its timing in code, and the client has its model and its grip in code of its own. We're going to move it into one items file, with a schema that gives each field a type, which both sides read, each taking the part it needs. The server is going to fight with the numbers from the file, and the client is going to draw the sword from it. And each player is going to hold a weapon, by its id, which the server saves and sends like the rest of the player. There's no inventory yet, so for now, that's the sword for all of our players. All right, let's get started with how MMOs keep their items.

Items in an MMO

2:10 An MMO keeps its items in two parts, like the NPCs in episode twenty-four. A template says what an item is, such as a sword's name, its model and its damage, and an instance says who holds one, by the template's id, along with what's particular to that copy. TrinityCore, the World of Warcraft server we looked at for the NPCs, keeps its templates in MySQL, in a table called item template. The game's client, however, comes with item files of its own, such as item dot d b c, where it looks up how an item is drawn. So adding an item to a private server means changing both, the table on the server and a patch for the players' clients, and keeping the two in step by hand. We write both the server and the client, so we can do better than that, with one file that both of them read.

2:59 There are a few ways we could keep the items in a file. A C++ table, like our lands and our monsters, has types that the compiler checks, but a change to the sword's damage means building and shipping both the server and the client again. JSON or YAML is easy to edit, and nlohmann's JSON library is in our build already, for Consul. However, neither of them has a schema that our C++ knows about, so we'd write the code that reads each field by hand, and a misspelt field would quietly fall back to its default. A MySQL table, like TrinityCore's, suits the server, while the client would still need a copy of its own, which brings us back to two. My proposal is protobuf's text format, which writes a protobuf message as text, with the fields by name, so that people can read it and edit it. Its schema is a proto file, like our nanommo dot proto, and protoc generates a C++ class from it, with a typed getter for each field. The parser refuses a field the schema doesn't have, or a value of the wrong type, and says on which line. And protobuf has been in our build since episode three, so the file costs us no new dependency.

4:09 Protobuf's documentation has a specification of the text format, which it suggests for configuration files and for tests. A message is a list of fields, each a name, a colon and a value, and a field that holds a message gets braces around its own fields, like this benchmark's input. A text format file holds a single message, and its extension is dot t x t p b, which the documentation asks for over the older ones. Two comments at the top of a file, proto file and proto message, name its schema, so that editors and tools that know the format can check the file, and complete its fields as we type.

One typed file

4:47 The items get a folder of their own, items, next to protocol and terrain, as both the server and the client are going to read them. In items dot proto, we use proto three syntax, like nanommo dot proto, in a package of its own, nanommo dot items, so that its messages don't mix with the protocol's. A weapon says how it fights, and its comment says whose part that is: the server's. Its reach is a float, with the unit in the comment over it, metres. The length of a swing is in milliseconds, with the unit in the field's name, as a two on its own, in the file, wouldn't say whether it means seconds or milliseconds. The hit is how far into the swing the blow lands, and damage is how much health a blow takes. A look says how an item is drawn, which is the client's part. Its model is the name of a glTF file in the client's weapons folder. The length is in metres, from end to end, and the grip is where the hand holds it, as a share of its length from the bottom. An item has an id, and the comment over it says that ids start from one, and that an item keeps its id for good, which we're going to need once ids go into the database and onto the wire. Then it has a name, a weapon and a look. Proto three keeps track of whether a message field is set, so an item that isn't a weapon, such as a potion later on, leaves the weapon out, and has weapon says so. Last, items is the whole file, with the weapon that a new player holds, by its id, and the items, as a repeated field.

6:15 The items go in items dot t x t p b, in the text format. The first two lines are the header from the specification, where proto file names our schema, and proto message says which of its messages the file holds, the items. Starting weapon is one, the sword's id, and the sword is the first item, in braces, with its id and its name. Its weapon has the numbers that the server has in code today, a reach of one point six metres, a swing of two thousand milliseconds, the length of the slash animation, a hit at nine hundred, and fifteen damage. Its look has the model's file, sword dot g l b, a length of a metre, and the grip a sixth of the way up, which are the client's numbers. So the file describes the sword as the two sides have it today, while each of them still uses its own constants.

7:05 Both sides read the file through a small library of their own, with its header in items dot h. In it, c s t d int, map, string and string view come in, along with Abseil's status or, for a table or an error, and items dot p b dot h, the C++ that protoc generates from our schema. An item table is the items in the file, by id, as a map from the id to the item's generated class. That way, finding a player's weapon is a lookup, without walking through a list, and the table keeps the starting weapon's id as well. Parse items takes the text as a string view, as our chat code does, and returns the table, or what's wrong with it. Its comment lists what can be wrong: a field that items dot proto doesn't have, an item with no id or with another's, or a starting weapon that isn't one. Read items reads a file from its path, for the server, the client and our first test, while the second test parses text of its own. In items dot c c, f stream and s stream come in, to read a file into a string, then string and string view. From Abseil, we take str cat, for the error messages, and from protobuf, tokenizer dot h, for its error collector, and text format dot h, for the parser. The parser reports each error to an error collector, a class with a virtual function that we override, record error. Without one, the parser prints its errors to the standard error stream, where a server's log would have them, while we'd like them in our status, so that the caller decides what to do. Record error gets the line, the column and the message. We leave the column's name commented out, as the style guide does for a parameter we don't use, which keeps the compiler from warning about it. Str append adds the line and the message to our text, with one added to the line, and the comment says why: the parser counts lines from zero, and editors count them from one. Text returns what we collected, as a const reference, so that reading it doesn't copy the string, and the string lives in the private part, as text underscore, which starts out empty. In parse items, the function that does the work, we make an empty items message, our collector, and a text format parser. We use the parser in place of text format's parse from string function, as it's the one that takes a collector. Record errors to hands the parser our collector, in place of the standard error stream. If the text doesn't parse, we return an invalid argument error with the collected text. That covers what the types can check, such as a misspelt field, or a word where a number goes. Then the table starts with the file's starting weapon, and a loop over the items checks what types can't. An item with the id zero has none, as zero is what proto three gives a field we left out, so the error names the item instead. Emplace puts the item in unless its id is taken, and tells us which in second, so false means that another item has that id already, and the error says which id. Last, we look for the starting weapon in the table, with find, and it has to be there, with a weapon part, or a new player would join holding nothing. Read items is what the server and the client call, with the path of the file. It opens the file with an input file stream, and returns not found, with the path, when it can't. It reads the whole file into a string stream, through the file's buffer, which reads it in one go. Then it parses the text, and puts the path in front of an error, so that a message says which file and which line went wrong.

10:37 The folder's CMake file makes the nanommo items library from the proto file and items dot c c, and runs protoc on the proto with protobuf generate, like the protocol library, into the build folder. The generated code's warnings are switched off, as it isn't ours to fix, and the library links protobuf and the Abseil pieces we use, publicly, so that the server and the client get them along with it. The tests are an executable of their own, with the path of the real items file passed in as a define, so that a test can read the file wherever the build runs. The top CMake file adds the items folder after protocol, and the API docs read it as well. In the Dockerfile, the lint stage checks the new folder's code with clang format and cpplint, along with the rest.

11:24 In items test dot c c, the first test reads the game's items from the real file, through the define. It's a test of our data as much as of our code, so a broken file fails in CI, before it gets anywhere near a server. Assert true stops the test if the file doesn't read, and prints the status, so we'd see the line that's wrong. Then it takes the starting weapon from the table, by its id, and checks that its name is Sword, that it does some damage, and that it has a model. It checks those loosely, so that tuning the sword's numbers doesn't break the test, while a file that doesn't parse does. The second test goes through what our checks refuse, one case at a time. A weapon with reach misspelt as r e e c h is refused by the parser, on line one. We compare the whole message, so that we see what a typo says: message type nanommo dot items dot weapon has no field named r e e c h. Two items with one id are refused too, with the text split over three string literals, which C++ joins into one. And a starting weapon of two is refused, as the file has a single item, number one.

12:34 We build it and run the items tests in Docker, through the compose service for tests, and both of them pass, so the sword reads from the file, and a typo stops the parse with its line.

The server's part

12:50 On the server, players fight in the world, so world dot h includes items dot h. A player gets a weapon, with a comment: what the player holds, by its id in the items file. We keep the id, and not a copy of the item, as the id is what we can save and send, and the table has the rest. The world's constants for the sword's reach, its swing and its hit go, as they're the sword's numbers, which the file has now. The world's constructor takes the item table after the seed and the terrain, with an empty one by default, so that the tests that walk around don't need one. Its comment says what that means: players fight with the items' weapons, so in a world without items, nobody can. Weapon of returns the weapon a player holds, as a pointer, which is null when they hold none, while swing and hit take the weapon as a const reference, which can't be null, as by then we know there's one. The world keeps a copy of the table, which is small, so each land has its own, without a pointer to something that could go away before it. In world dot c c, the constructor moves the table into the world. In join, a player whose weapon isn't in the table gets the starting weapon, and the comment covers both cases: no weapon, or one that isn't in the items file. Such a player is joining for the first time, or holds a weapon we took out of the file. In act, we look up the player's weapon once a tick. A player who's swinging has a weapon, as a swing is how they got there, so the hit comes from its milliseconds, as a chrono duration, which compares with the swing's nanoseconds without us converting anything. The blow lands with the weapon, and the swing ends after the weapon's swing time. A player who holds nothing doesn't start a swing, and moves instead. Weapon of finds the player's item in the table, by its id, with find, which returns the end of the map when there's no such id, as for a player in a world without items. If there's no such item, or the item isn't a weapon, such as the potion we talked about, it returns null. Otherwise, it returns the address of its weapon part, which lives in the table, so the pointer stays good as long as the world does, and nothing gets copied on each tick. Swing looks for the nearest monster within the weapon's reach, and hit checks the reach again when the blow lands, as the monster may have walked off in the meantime. The damage is the weapon's, plus what the player's level adds, with a cast to int, since the file's damage is unsigned and health is an int, and mixing the two in a subtraction would go wrong below zero. So in levels dot h, damage becomes extra damage, with a comment: what a level adds to the damage of the weapon a player holds. In levels dot c c, that's five for each level after the first, as before, while the fifteen comes from the sword now.

15:30 The server's library links nanommo items, next to the protocol and the terrain. In server dot h, items dot h comes in, and create takes the item table, after the seed. The server doesn't keep the table, as it hands each land a copy, so create takes it as a const reference, which doesn't copy it on the way in. The constructor gets it too, and clang format moves the arguments around to fit each line. In server dot c c, create's definition gets the table, and passes it on to the constructor, where the call is laid out again by clang format, as it's grown past eighty characters. We could have passed a pointer to the table, but a const reference can't be null, so a server can't be made without its items, and it can't change them either. The constructor's definition gets the table as well, after the seed, and doesn't keep it. Each land is a world of its own, which the constructor makes in a loop over the lands, from episode twenty-two. In that loop, each land's world gets the table, after its seed and its terrain, and keeps a copy. In main dot c c, items dot h comes in. After the seed, the server reads the items, from the nanommo items setting, or from items slash items dot t x t p b by default, a path that works when the server starts in the repo. If it can't read them, it prints why and stops with one, like it does for a bad setting, as a world server without its items can't run a fight. Then create gets the table, from the status or, with a star.

In Docker

17:02 Let's build it and run it the way our world servers run, in Compose. Docker compose up builds the server's image, which runs our tests on the way, and starts the server with MySQL, Redis and Consul, and the server exits with an error. Its log says why: not found, couldn't read items slash items dot t x t p b. The image has the server's binary from the test stage, and nothing more. The items tests ran there, next to the whole repo, so it's easy to miss that a data file now has to ship with the binary.

17:34 In the Dockerfile's server stage, we copy the items file into slash u s r slash local slash share slash nanommo, which is where Linux keeps the read-only data of programs installed by hand. An ENV line sets the nanommo items setting to that path, so that the server finds the file whatever folder it starts in. The profiling stage from episode eight runs the server too, so it gets the file and the setting as well.

18:01 We build the image again and start the server, and this time it stays up. Its log says it's listening on port seven seven seven seven, at twenty ticks a second, which means it read the items before it got there.

18:15 Our world tests make worlds without items, which is fine for walking, while the two tests that swing need a weapon. In world test dot c c, items dot h comes in, and sword makes the tests' own item table from text, with parse items. Its comment says why the tests have their own: so that the game's sword can change without breaking them. The text is a raw string, like the ones in our earlier tests, and the p b in front of the bracket tells clang format that it's a text proto, so it lays the text out too. It holds a sword with the reach, the times and the damage that these tests were written for, on a single line, and without a look, as the server doesn't draw anything. Swings at the nearest monster in reach gets a world with the sword, after a seed of one and the default terrain, and so does kills monsters for their experience. Their checks stay as they were, such as fifty minus fifteen health after a hit, as a sword from a file should hit like the sword from the code did.

19:13 We run the world tests, and they all pass, the ones that walk without items, and the ones that fight with the tests' sword.

The client's part

19:36 The client reads the items file too, from a copy that the build puts next to it, like the models. What it needs from the server is the weapon each player holds, by its id, so that it draws the right model in each player's hand. A weapon changes about as often as a name, so it goes out with the names and the levels, in the full world, once a second, and the client keeps it in between, for two bytes a player each second. Protobuf leaves out a field that's zero, so the worlds in between carry no weapon at all. If zero were an item's id, the client couldn't tell that item from a weapon that wasn't sent, which is why our ids start from one. The client loads each item's model once, as it starts, at the length and the grip from its look, and draws each player with the weapon they hold.

20:24 In nanommo dot proto, the player gets weapon, field number ten, an unsigned int, with a comment: what the player holds, by its id in the items file. In server dot c c, to protocol copies the player's weapon into the message. In broadcast, the comment gets weapons, next to the names, the levels and the most health a unit can have, which change now and then, so they go out once a second, and the clients keep them in between. On the worlds in between, the weapon is cleared, along with the name and the level. In parts dot c c, type traits comes in, for a check on the unit's type that we're going to need in a moment. Update keeps the last name and level when a world leaves them out, and its comment gets weapons too. Protobuf leaves out a field that's empty or zero, which a name, level or item id can't be, so a zero means that this world didn't send it. Update is a template, for players and for monsters, and a monster has no weapon field, so the weapon goes in an if constexpr. The compiler works out its condition as it builds each version of the template, and leaves the block out of the monsters' version, where the weapon wouldn't compile. The condition, from type traits, is true when the unit is a player. A player whose weapon wasn't in this world keeps the one we had. In parts test dot c c, the test that keeps names and levels between the ticks that send them gives the first player weapon one in the full world, and checks that it's still there after the compact world.

21:54 In the client's CMake file, the client's library links nanommo items, and the target that copies the assets next to the client copies the items file too, with copy if different, so that a build that changed nothing copies nothing. So the client finds the file there, whichever folder we start it from. In animator dot h, a figure gets the weapon, by its id in the items file. And in animator dot c c, update copies each player's weapon into their figure, from the world that parts put together, where it stays between the full worlds.

22:27 In characters dot h, the optional header goes, and c s t d int and map come in, with items dot p b dot h, for the look. Items file returns the path of the items file next to the client, like the folders of the models. Hold takes an item's id and its look, in place of a path and a length, as the look has the file, the length and the grip together. Hold is called once for each item, as the client starts, and the comment over it says what it does now: it loads the item's model, to draw in the hand of each figure that holds the item. A weapon, in the private part, is a model and its grip, the matrix that puts it in a hand, and the characters keep one for each item, by id, where they had the one sword. In characters dot c c, k grip goes, the sword's grip as a constant, with its comment, as the look has it now. Items file is the application's folder, from raylib, and items dot t x t p b. The destructor unloads each weapon's model, going through the map with a structured binding, as before. Hold builds the path from the weapons folder and the look's model, and skips an item with no model, or one whose file isn't there, in which case the player is drawn with nothing in their hand. The scale comes from the look's length, a metre for the sword, over the model's height, and the grip comes from the look too, as a share of the model's height. The turns that put a blade along the knuckles stay in code, as they're about Mixamo's hand, whatever it holds. Last, the model and its grip go into the map, under the item's id. In draw, we look up the weapon the figure holds, and draw nothing in its hand when it holds none, or one we don't know, such as an item from a newer file on the server. Otherwise, we take that weapon's model, and put it in the hand with its own grip, as each weapon is held its own way. In renderer dot c c, items dot h comes in, and the constructor reads the items from the file next to the client, and has the players' characters hold each item, with its look. The comment over the loop says why they're drawn with the cube's shader: weapons have no bones. If the file doesn't read, the client says why, with raylib's trace log, and draws nobody holding anything, as it can still show the world.

24:41 Let's build it and run it, and as nothing saves a weapon yet, alice joins with the starting weapon. Once she's in, she holds the sword, which the client loaded from its copy of the items file, a metre long, and held a sixth of the way up. Turning the camera around her shows it in her hand, along her knuckles, the way it was when it came from code. And a swing, with Space, takes the two seconds from the file, on the server.

25:08 We run the protocol tests for parts, and they pass, with the weapon kept between the worlds that send it.

Saving the weapon

25:18 A player's weapon is part of what we keep of them, like their land, so it gets a column. The migration is V five, players weapon, and it adds the weapon column, an int unsigned for an item's id, which can be null, with algorithm instant, like our other columns. Null is a player we haven't saved a weapon for, whom the server gives the starting weapon, so the database doesn't need to know which item that is. In players dot h, a player's progress gets the weapon, and in players dot c c, load selects it, and reads it from the player's hash in Redis, as the hash is newer. Like health, an empty value means none saved, which the world takes as the starting weapon, as the comment says, while a value that isn't a number is an error. Save writes the weapon into the hash with the rest, and flush reads it back out of the hashes, for both kinds of key, and writes it into the weapon column. A hash saved before this episode has no weapon, which goes in as null, like a missing health, and the comment says so. On a duplicate key, the row's weapon is updated with the rest. In server dot c c, progress of copies the player's weapon, for the dead and for the living. Nothing changes a weapon yet, so after the first save, which writes the starting weapon, a weapon doesn't make a player count as changed.

26:32 In players test dot c c, loads where the player was saved in that world saves a weapon with the later progress. With the weapon, the progress no longer fits on a line, so clang format lays it out a field to a line. The place, the experience, the health and the land stay as they were, and the weapon is one, the sword, which load should give back from the player's hash in Redis. Then the test checks that the weapon comes back when the player loads. Flushes the players to MySQL saves weapon one in Frost, with the rest of that player's progress, the place, the experience, the health and the Ashlands. After the flush, the test selects the weapon column from the players table too, and expects a row for each of the two players it saved. The player saved before worlds has a hash with no weapon, which flush writes as NULL, and our database code reads a NULL back as an empty string, and one for the player in Frost.

27:23 The player tests need MySQL and Redis, so this time Compose starts what the tests depend on: MySQL, with Flyway running our migrations, V five among them, and Redis. They pass, so a weapon goes into Redis, comes back out, and reaches the players table.

Docs

27:40 Last, the docs, where the weapons' README says that the items file sets how long the client draws the sword and where the hand holds it. The numbers it gave, a metre long and held at the hand, are the file's now. The main README gets a new section on items, which says that they're data, in one file that the world server and the client both read. The file is in protobuf's text format, with its schema next to it, and the section says which part of an item each side reads. It says what the parser checks and what our own checks add, why ids start from one and stay with their item, and how a player's weapon is saved and sent. It ends on where each side finds the file, and on building the server and the client from one commit, as a client whose file doesn't have a weapon draws the player holding nothing. The characters and fighting sections say that the sword's numbers come from the file, saving lists the weapon in the player's hash, and the table of the worlds' settings lists nanommo items, with its default.

Wrap-up

28:40 So, today our sword moved out of code, into one items file in protobuf's text format, with a schema that both the server and the client compile in. The parser refuses a field that the schema doesn't have, and our own checks refuse an item without an id, two items with one id, and a starting weapon that isn't one. The server fights with the weapon's numbers from the file, which its image has to carry, as we found out when the server stopped in Compose. The client reads its own copy of the file, and draws each weapon from its look. Each player holds a weapon by its id, which goes out once a second, and into Redis and MySQL with the rest of the player.

Playing it

30:10 Let me know in the comments what you'd like to see next. And if you want to follow along with the series, subscribe to the channel. I hope you find this useful, and thanks for watching.