MMO NPC with an LLM in C++: threads, batching

Intro

0:00 Hello and welcome back to episode twenty-five, where one of our NPCs is going to tell our players stories, from a few words they give it, written by a small language model. Let's begin with a quick look at where we're going to be by the end of the episode.

0:14 Alice logs in, and starts in the middle of the Meadows, near our NPCs. She walks up to the storyteller, the purple ball across the road from the guard, clicks it, and gives it three words. A small language model writes her a story with them, on a server of its own, and the story comes back in her chat, under the storyteller's name.

0:39 On the website of the series you can find more information about the project. The code for the project is open source on GitHub, tagged at each step of each episode so that you can follow along.

Last time

0:51 Now, before we get to today's work, let's quickly go over what we did last time. Our monsters became data, with a kind for the skeleton, and tables of where monsters spawn and camp in each land. Then our first NPCs got tables of their own, a guard, a merchant and a healer in each land, with an ash warden in the guard's place in the Ashlands. On the server, an NPC is a small type of its own, with ids of its own, apart from the monsters, which goes into its interest cell once, and out to the players once a second, with its look. The client keeps each NPC for three seconds after it last heard of it, so that a lost world doesn't make them blink out, and draws them as coloured balls with green names, until they have models.

Today

1:33 Today, one of those NPCs is going to do something: a storyteller, which writes you a short story from a few words you give it, with mvGPT, the small language model from my other series. mvGPT's server goes into our repo, and runs in Compose next to the game. A player's words for an NPC travel with their input, and the server checks them before the model sees them, then asks for the story from threads of its own, so that the tick doesn't wait for the model. On the client, a click on the storyteller opens a window for the words, and last, the dashboard counts the stories, and our bots ask for them too. All right, let's get started with where the stories come from.

mvGPT

2:10 mvGPT, a minimum viable GPT, is my other series, where we write a language model from scratch in PyTorch, and take it through the whole pipeline of training one, from pretraining to fine-tuning, and on to serving it. Its code is open source on GitHub, tagged at each step like ours, and the models it trains are on Hugging Face, where anyone can download them.

2:33 Its README says what it is: a small transformer, like GPT's, written from scratch in PyTorch, which learns to write children's stories from TinyStories, a dataset of short stories in simple English. It says how to run it, and where the docs are, which go through each part. The model is small, so it writes a story on a CPU, which is what lets it run next to a game server, on the kind of machine our servers run on.

2:57 On the Hugging Face Hub, the model's page is its model card, with how to use it with the transformers library. Each episode of mvGPT that trains a model pushes it here under a tag, and we're going to use ep003-sft, the fine-tuned model from its third episode, which learned to write a story from a few words. In mvGPT's episode fourteen, we built a server for it, which writes the stories of many players at once, and that's the server we're going to run. Its limitations say what it doesn't do: it writes children's stories, and nothing else, so no facts, no arithmetic and no chat.

Why a storyteller

3:35 mvGPT learned from TinyStories, short stories written in simple English, for children, and its fine-tuned model writes a story from a few words. That's what it does well, and as its model card says, it writes nothing else, so an NPC that holds a conversation with you is out of its reach. Its vocabulary is ten thousand words, from those stories, so it knows a dragon, a castle and a sword, but not a warden, a healer or a merchant, which reach it as unknown words. So my proposal is an NPC that plays to what the model does well: a storyteller, which takes a few words from you, and tells you a story with them. An NPC could chat with a hosted model, such as OpenAI's, which we'd pay for with each request, and which sends what our players type to another company. Or it could chat with a bigger model that we run, which needs a GPU for each world to answer quickly. A small model on a CPU keeps it all in our Compose, for the cost of a container.

4:33 Let's follow a story from one end to the other. Alice clicks the storyteller, and types a few words into its window, which go to the server with her input, in a new field, talk, along with the storyteller's id. The server checks that she's close enough to talk to it, cleans her words like a line of chat, and makes sure that she isn't asking for stories too often, as each one keeps the model busy for a while. Then it builds a prompt in the shape that the model learned its stories from. A thread of the server's, apart from the tick, posts the prompt to the storyteller's completions API, over HTTP, on Compose's network, where the storyteller writes it along with the other prompts that are waiting, as a batch. When the story comes back, the server sends it to alice and nobody else, as lines of chat under the storyteller's name, over TCP.

The storyteller in Compose

5:19 We start a folder for the storyteller next to our server's code, and copy mvGPT's files into it, from the end of its episode fourteen. mvGPT comes under the MIT licence, which asks for its notice to go with any copy, so its licence comes along, and a short README says where the files come from, and what they serve. The server needs a single Python package on top of PyTorch, transformers, from Hugging Face, which downloads the model from the Hub and runs it. Tokenizer dot py is mvGPT's word-level tokenizer, which the server uses to turn the model's tokens back into text, with the spaces in the right places. And server dot py is the server we built in that episode, with Python's own HTTP server, which answers each request on a thread of its own, and one more thread for the model. Each request waits in a queue until the model's thread has written its story. The model's thread takes the first request in the queue, gives others five milliseconds to join it, up to sixteen, and writes their stories together, as a batch. That way, sixteen players who ask at once share the model's work, without waiting in line one after another.

6:27 mvGPT's Dockerfile has a stage for the server, which becomes a Dockerfile of our own, in the storyteller's folder. It starts from Python's slim image, which is Debian with Python and little else, and works in a folder called app. It installs PyTorch's build for the CPU, from PyTorch's own index of builds, pinned to the version that mvGPT was built with, and without pip's cache, so that its downloads don't stay in the image. The comment over it says why it's that build: the server has no GPU, and the CUDA build is gigabytes bigger. Host is zero dot zero dot zero dot zero, so that the server listens on all interfaces, as a published port can't reach it on localhost inside a container. Then the requirements file, which pip installs transformers from, and the two Python files, the server and the tokenizer it decodes with. It exposes port eight thousand, and its command starts the server with Python, with the defaults we gave it in mvGPT: batches of sixteen, and five milliseconds to wait. In compose dot yaml, the storyteller is a service, built from that folder. The Hugging Face library keeps what it downloads in a cache, in root's home folder, which goes in a volume called huggingface, so the model downloads once, and the next container finds it there. Port eight thousand is published, so that we can try the server from our machine, and the health check asks the server for slash health, with Python, as the image has no curl. Start period gives it five minutes before a failed check counts, for that first download. In production, its port goes, as the world servers reach it on Compose's own network, and it restarts unless we stop it, like the others. The game's own Docker builds don't need the storyteller's files, so the ignore file leaves them out, and CI builds the storyteller's image along with the other images.

8:12 Compose builds the storyteller's image, which has PyTorch in it, so the first build takes a while, and starts it. Dash dash wait waits for the health check to pass, which takes as long as the model takes to download, the first time. Then curl posts a prompt to slash v one slash completions, in the shape of OpenAI's API, with a hundred tokens at most. The prompt is words, dragon, castle and gold, and then story, with a colon, on a line of its own, which is how mvGPT's fine-tuning showed it a story's words. Jq prints the JSON reply so that we can read it, and the story is in choices, under text. Finish reason says length when the model used up the hundred tokens, and the story stops in the middle of a sentence, or stop when it ended the story on its own. Usage counts the tokens, nine in our prompt, and what the story took, each a word or a punctuation mark, as mvGPT's tokenizer works by words.

Talking to an NPC

9:10 A player is going to talk to an NPC by sending it words, so in our proto file, a new message, Talk, has the id of the NPC, and the words, as a string. Its comment says what it is: what a player says to an NPC that talks, by the NPC's id. It goes in the input, as its sixth field, talk, next to say, as talking to an NPC is something a player does, like saying a line, and it travels over TCP with the rest of the input. A message field that isn't set takes no bytes, like an empty say, so the inputs we send while walking stay as small as they were. The NPC message gets a bool, talks, so that the client knows which NPCs to open a window for, with no table of its own, like their look. In world dot h, the server's NPC gets talks too, for whether players can talk to it. In npcs dot h, talks is part of each kind too, and the comment over it adds what talking means for now: an NPC that talks tells stories. So the storyteller is a new kind, called Storyteller, which is the name that players see over it. It gets a purple of its own, apart from the blue guard, the gold merchant and the green healer around it, a radius of half a metre, like the merchant's, and talks set to true. Like the other kinds, it's a constexpr with designated initialisers, so each field is named where it's set, and a kind that leaves talks out doesn't talk, as the field starts as false. The comment on where they stand becomes a guard and a storyteller on either side of the road to the gate, facing the players, and a merchant and a healer facing each other across it. The Meadows' storyteller stands across the road from the guard, and the Ashlands' across the road from the ash warden, so the table has eight NPCs now. In npcs dot c c, spawn NPCs copies talks from the kind into each NPC it spawns.

10:56 The server builds the prompt that the model gets, from the words, in a new header, stories dot h, as both the server and a test are going to need it. It has cstddef, optional, string and string view, for the types in its two declarations. K story words is three, which is how many words mvGPT learned to write a story from. Each story in its fine-tuning came with three words to use. Story prompt takes the words as a string view, and returns the prompt, or nothing when there are no words, as an optional, like clean chat does. Its comment says what the prompt is: mvGPT's prompt for a story with the words in it, the way it learned them. Its fine-tuning gave the model a story's words after the word words and a colon, with commas between them, and on the next line, story, with a colon, after which came the story. So a prompt that ends on story and a colon has the model go on with a story, the way it did many times while it learned, with our words in it, as best it can. The comment goes on with what the function does with what the player typed. It takes the first three words, split at spaces and commas, in lowercase, and returns nothing if there are none, so that the server doesn't ask the model for a story about nothing. In stories dot c c, our header comes first, then the standard headers, and Abseil's headers for ASCII, which has its lowercase, and for str cat, str join and str split, which build strings and break them up. We keep the words in a vector of strings, as each one is going to be a lowercase copy. Abseil's str split splits the text at any space or comma, skipping the empty pieces, so that a comma and a space after a word, or two spaces, don't make empty words. Once we have three, we stop, as a player can type more, and each word goes in lowercase, as the stories it learned from give their words in lowercase. The model's vocabulary keeps Dragon, with a capital, apart from dragon. With no words, there's no prompt, and otherwise, str cat puts words in front, the words joined by a comma and a space, and story after a new line. The server's library gets stories dot c c, and its tests get stories test dot c c, which we write in a moment.

13:08 In server dot h, talk takes the player's id and their talk message, and notify sends the player a line from the server, which has no name, as its comment says. We wrote that for chat already, and stories are going to need it a few times. Story limits keeps a token bucket for each player, like chat limits, for how often they can ask for a story. In server dot c c, cmath comes in, for hypot, along with Abseil's str cat, and stories dot h. K talk reach is four metres, from the middle of the NPC, about as far as you'd stand to talk to someone. K story burst is one, so the bucket for stories holds a single story, and a player can't save stories up to ask for several at once. Over it, we write why stories are much stricter than chat: a story is a hundred tokens of the model's work, where a line of chat costs nothing. K story refill is ten seconds, in chrono seconds like the chat's two, so a player gets one story each ten seconds. An NPC's talks goes out with it, and an input with a talk in it goes to talk, as has talk says whether the field was set. A player gets a full bucket for stories when they join, and it goes when they leave. Say's notice for a player who talks too fast becomes a call to notify. Talk starts from the player's land, which land of finds by their id, and the player in it, for where they stand. It looks the NPC up in that land's NPCs, as each land counts its NPCs from one, so an id names a different NPC in the other land. For an NPC that isn't there, or doesn't talk, the player gets nothing back, as our client doesn't send that. Then the distance from the player to the NPC, on the ground, with hypot, and a player further than four metres gets a notice that they're too far away to talk to the NPC, by its name. The server decides that, as it knows where everyone is, and a client could send any id from anywhere. The words go through clean chat, like a player's line in chat, and story prompt makes the prompt from what's left, and without a prompt, there's nothing to ask for. The bucket comes last, so that words that go nowhere don't use up a story, and a player who asks too soon gets a notice that the NPC needs a moment before the next story. For now, the log says who asked which NPC for a story, and with what words. Notify is the code from say: a world with a single line of chat with no name in it, sent to that player's session, framed like the other messages that go over TCP.

15:38 A test pins the prompt down, in stories test dot c c. The model wouldn't tell us that it got a prompt in the wrong shape, it would write a worse story, so a test is the place to catch it. It includes our header, optional and GoogleTest, and the test goes in an unnamed namespace, inside ours, like the chat's tests. Prompts with three words goes through the cases, a line each. Dragon castle gold, with a capital, comes out in lowercase, after words and a colon, with commas between them, and story on the next line, which is the prompt we sent with curl. Sword, skeleton, brave and gate, with extra spaces around a comma, gives the first three, and gate goes, while a single word makes a prompt too, as the model can write a story about ash. And a space and a comma make no prompt at all, as there's no word in them to tell a story about.

16:29 We build the tests, and run the server's with a filter for the stories, which need no database, so the flag for no dependencies keeps Compose from starting one, and the prompt's test passes.

Off the tick

16:45 With a prompt, the server has to ask the storyteller for a story. Our world runs twenty ticks a second, so a tick has fifty milliseconds for all its work, while a story takes the model about a second to write, and longer when it's busy. If the tick waited for it, the whole world would stop for a second, for each story that anyone asks for. We met that with Consul, in episode twenty-one, where the heartbeat goes to a thread of its own, so that a slow answer can't hold up a tick. A single thread for the stories would keep the tick going, but it would ask for one story at a time, and the storyteller writes the requests that are waiting together. With one thread, there would be one request waiting, and each of its batches would be a single story. So the storyteller's client gets sixteen threads, as many as the storyteller writes at once, and each one waits for one story. When a story comes back, it goes to the server's own thread, which runs the ticks and has the players and their sessions, so that those don't need a lock.

17:42 The storyteller's client gets a header of its own, storyteller dot h, with cstdint for the port, functional, for the callback, string, Abseil's status or, as asking can fail, and Asio's io context and thread pool. Its comment says what the class does: it asks mvGPT's completions API, the storyteller service, for stories, on threads of its own, so that a tick doesn't wait for the model. K port is eight thousand, the storyteller's, like Consul's port in its class, and told is the callback's type, a std function that gets the story, or why there isn't one, as a status or of a string. The constructor takes the io context that the server runs on, which is where the stories go back to, and the storyteller's host. Tell asks for a story that goes on from the prompt, and calls told with it on the io context's thread, where the rest of the server runs. Tell returns straight away, and the story comes later, which is what lets the tick carry on. Complete is the request, which runs on one of the threads and blocks it, and it's const, as it changes nothing in the class. Threads is an Asio thread pool, like the one that main has for Consul, which runs what we post to it on its threads. In storyteller dot c c, httplib, which we've had since Consul, then string and utility, Abseil's str cat, Asio's post, and nlohmann's JSON, for the request and the reply. K threads is sixteen, as the storyteller writes the stories that are waiting together, up to sixteen, and each of these threads waits for one story, so with fewer, it would write fewer at once. A thread that waits for a reply costs a little memory for its stack, and no CPU, so sixteen of them are cheap. K max tokens caps a story at a hundred tokens, which is a few lines of chat. The timeout is ten seconds, as a story takes a second or two, and longer while the storyteller is busy, and after ten, the player has stopped waiting. The constructor keeps the io context and the host, and makes the pool with sixteen threads, which start straight away and wait for work. Tell posts a lambda to the pool, which captures this, to call complete, and moves the prompt and the callback into the lambda, so that the thread has them for as long as it needs them, after tell has returned. The lambda is mutable, as moving the callback out of it again, further down, changes it. On the pool's thread, it completes the prompt, which blocks that thread until the story comes, or the request fails, and then it posts another lambda, with the callback and the story moved in, to the io context. That lambda calls told with the story, so told runs on the server's thread, between ticks, like the handlers of our sockets. Complete makes an httplib client for the storyteller's host and port, like Consul's, with a second to connect, and ten seconds to read the reply. The request is JSON, with the prompt and max tokens, which it posts to slash v one slash completions, as application slash json. If the request didn't get there, such as when the storyteller is down, it returns unavailable, with httplib's error as a string. An answer that reached us, but isn't two hundred, such as a prompt that's too long for the model, is an internal error, with the status and the body, which has the storyteller's message. It parses the body without exceptions, the false at the end, as the style guide doesn't use them, so a reply that isn't JSON parses as discarded. A JSON pointer finds the text of the first choice, where we saw it in curl's reply. A reply that isn't an object, or has no text there, or text that isn't a string, is data loss, as the reply lost the story we asked for, and otherwise, the text is the story. The server's library gets storyteller dot c c, after stories dot c c.

21:24 The server gets the storyteller the way it gets its metrics and its players, from main. In server dot h, we include storyteller dot h, and create takes a reference to a storyteller, after the players, and so does the constructor. The server keeps it as a reference, as main owns the storyteller, which outlives the server. In server dot c c, create passes it on to the constructor, in the line that makes the server, and the constructor keeps it, after the players. The list follows the order of the members in the class, as C++ initialises members in that order, whatever order the list is written in, and the compiler warns when they differ. At the end of talk, after its checks and its log line, we ask the storyteller to tell the story, with the prompt, and a lambda to run when the story comes back. It captures this, for the sessions, the player's id, and the NPC's name, as a copy, so that the lambda holds nothing that points into the land's NPCs. By then, the player may have left while the model wrote, as the comment says, so we look their session up again, and a player who left gets nothing. The text starts as the NPC saying it can't think of a story right now. With a story, it's the story, cleaned like a line of chat, as it comes from outside the server, and could have anything in it. Without a story, the server logs why, such as the storyteller being down, and a text that cleans to nothing goes nowhere. The story goes out as a world with a line of chat in it, under the NPC's name, to this player alone, over TCP, as a story is for whoever asked for it. In main, we include storyteller dot h, and make the storyteller next to the io context, from the setting nanommo storyteller host, which is our machine by default, and pass it to create. In compose dot yaml, the world servers find it at storyteller, the service's name on Compose's network.

Clicking the storyteller

23:17 To click the storyteller, the client has to work out what's under the mouse. Raylib's get screen to world ray turns the mouse's position into a ray from the camera: a point, o, where the camera is, and a direction, d, through the mouse. So a point on the ray is o plus t times d. The direction has a length of one, so t is a distance, in metres. Our NPCs are balls, and a point on a ball is r away from its middle, c. The ray touches the ball where both are true, and putting the first into the second gives a quadratic in t. The ray hits the ball when the quadratic has a real answer of zero or more, and the smaller answer is how far away the hit is. Raylib's get ray collision sphere works that out for us, and gives back whether it hit, and the distance, so when one NPC stands in front of another, we pick the nearest.

24:07 On the client, a figure gets talks, for whether the player can talk to it, by clicking it. The animator copies it from the NPC in stand, along with its colour and radius, as the server sends it. Picking gets a header of its own, picking dot h, with raylib, for its ray, vector, and the animator's figure, as the animator's figures are what the renderer draws, with where each NPC stands on the ground. Pick NPC takes the figures and a ray, and returns the nearest NPC along the ray that the player can talk to, such as the one under the mouse, or a null pointer if there's none, as its comment says. A pointer into the vector says which figure without copying it, and it stays good until the figures change, at the next frame. It takes a ray, and not the mouse, so that it doesn't depend on the window, and a test can hand it any ray it likes. The window and the camera stay in main, which turns the mouse into a ray. In picking dot c c, its own header comes first, then limits, for infinity, and vector. Pick NPC starts with nothing picked, a null pointer, and goes through all the figures, which is cheap for a click, even with hundreds of players around, as most of them are skipped straight away. We keep the nearest hit so far, which starts at infinity, so that the first hit is nearer. We skip any figure that isn't an NPC that talks, so that a player or a monster in front of the storyteller doesn't get in the way of the click. The ball rests on the ground, so its middle is a radius above where the NPC stands. Get ray collision sphere does the maths that we just saw. A hit that's nearer than the nearest so far becomes the nearest, and the one we pick, and after the loop, we return what we picked, which is still a null pointer if the ray hit nothing. In input dot h, clicked world says whether the left button went down this frame on the world, and not on one of the GUI's windows, as its comment says. In input dot c c, that's ImGui not wanting the mouse, which it does over its windows, and raylib's is mouse button pressed, which is true in the frame the button goes down, so that a click picks once.

26:09 Then the window that takes the player's words, in a new header, talk window dot h. Array is for the buffer we type into, cstdint for the NPC's id, optional and string, chat dot h for the longest line, and our generated protocol header, for the talk message that the window hands back. Talk window is a class, like the chat window, as it keeps what's typed between frames, and which NPC it's for. Its comment says it's a window to talk to an NPC, such as the storyteller, which opens when the player clicks it. Open takes the NPC's id and its name, for the title, and build draws the window while it's open, and returns what the player says to the NPC, once they ask, and closes it. The NPC's id is an optional, which is empty while the window is closed, so that it doesn't need a flag of its own for that. Focus says that the field should take the keyboard on the next frame, and words is the buffer for ImGui's field, one char longer than the longest line, for the zero at the end, like the chat's. In talk window dot c c, its header, imgui, cfloat for the smallest float, the standard headers, and the Font Awesome icons that rlImGui comes with. Open keeps the NPC's id, moves the name in, and sets focus, so that the window takes the keyboard as soon as it appears. Build draws nothing, and returns nothing, while there's no NPC, which is most frames. The window goes in the middle, three quarters of the way down, with a pivot of a half, so that its middle sits there. That puts it below the NPC we clicked, which is higher up on the screen, as the camera looks down at the player from behind. Cond appearing places it there each time it opens, and leaves it where the player drags it while it's open. It's three hundred and sixty pixels wide, and a height of zero has ImGui fit it to what's in it. Open goes to begin, which puts a close button in the title bar, and sets open to false when it's clicked. The window's title is the NPC's name, and it doesn't resize or collapse, as there's nothing in it to make room for. It says give me a few words, and I'll tell you a story. On the frame it opens, set keyboard focus here puts the keyboard in the field under it, so that the player can type straight away, without a click on the field. The field says dragon, castle, gold, in grey, while it's empty, as an example, and Enter returns true, like the chat's. The button has an open book from Font Awesome, and says tell me a story. Enter or the button, with something in the field, makes the talk, with the NPC's id and the words, empties the buffer for next time, and closes the window, as the story is going to come back in the chat. After end, a window that's closed, by the button in its title bar or by asking, forgets its NPC. The client's library gets talk window dot c c and picking dot c c, and its tests get picking test dot c c. In main, we include picking dot h and talk window dot h, and make a talk window next to the chat window. After the figures are updated, a click on the world makes a ray with raylib's get screen to world ray, from the mouse's position and the camera. The NPC it picks, if any, opens the talk window, with its id and its name. In the GUI, after the chat window, which sends what we say in the chat, the talk window is drawn, while it's open. When build returns a talk, the talk goes to the server in an input, a copy of the last one we sent, like the chat's, so that we keep moving as we were.

The first story

29:37 Let's build it and run it, with the login server and Ember behind the client, and the storyteller in Compose. Alice creates her account, and starts in the middle of the Meadows, with the storyteller up ahead, the purple ball across the road from the guard. A click on it opens its window, with the keyboard in the field, so she types dragon, castle, gold, and presses Enter. The server's yellow line says that she's too far away to talk to the storyteller, which is right, as she's about eight metres away from it. So she walks up to it, clicks it again, and asks for her story. The window closes, and we wait, and no story comes back, and nothing in the chat says why. A second try gets a yellow line that says the storyteller needs a moment before the next story. So the server took her words the first time, and asked the storyteller, and the story got lost somewhere on its way back to her.

30:31 The clicks work, so before we go looking for the story, a test pins the picking down, in picking test dot c c, with cstdint, vector, GoogleTest and our header. A helper in an unnamed namespace makes an NPC's figure, with an id, a place along z, and whether it talks, as a ball of half a metre, standing at a height of zero. Picks the nearest NPC that talks lines three of them up along z: the first, ten metres away, talks, the second, at five metres, doesn't, and the third, at seven, talks. A ray from the middle, at the height of the balls' middles, looks along minus z, through all three of them, and another, two metres to the side, looks past them. The first ray picks something, as an assert, so that the test stops there instead of reading a null pointer, and what it picks is the third, the nearest that talks, through the second, which doesn't. And the ray two metres to the side misses them all.

A story in lines

31:27 The storyteller wrote a story, as curl showed us it does, and the story reached the server, which put it through clean chat before sending it. Clean chat returns nothing for a line longer than two hundred bytes, and a story of a hundred tokens is about twice that, so the text came back empty, and the story went nowhere, without an error. The limit was made for what a player types in a line, and a story is a few lines' worth. So, in stories dot h, a new function, story lines, breaks a story into lines of chat, with vector for the lines. Its comment says how: each line is at most k max chat length, broken after the last sentence that fits, or between words in a sentence that doesn't. A story that ran out of tokens stops in the middle of a sentence, so the story ends at its last full stop, or question or exclamation mark, when it has one. We could raise the limit for NPCs, but a line of four hundred bytes fills the chat window, and lines that end on whole sentences read more like someone telling a story. In stories dot c c, algorithm, for min, and chat dot h, for the longest line. An unnamed namespace holds a constant and a helper that no other file needs. K sentence ends holds the three marks that end a sentence, as a string view, and ends sentence says whether a word ends one. Above it, we write how: with one of those, or with one of those and the quote that closes what someone says, as children's stories are full of people talking, and the model learned to write their quotes. A word that ends with a quote loses it, and what's left ends a sentence when its last character is one of the three. Story lines finds the last of those marks in the story, and takes a closing quote after it, as a sentence can end inside the quotes of what someone says, so the cut keeps it. Then it cuts the story there, as a string view, which doesn't copy anything. Then it builds the lines word by word, with the line so far, and sentences, how much of the line is whole sentences, so that we know where to cut it. It splits the story at any whitespace, so a new line in the story goes too, and skips the empty pieces. While the next word, with a space before it, wouldn't fit, the line is cut, after its whole sentences when it has any, or else all of it, and the part before the cut becomes a line. What's after the cut, past its space, starts the next line. It's a loop, as what's left after the last sentence can still leave no room for the word. A space goes between words, then the word, and a word that ends a sentence moves sentences to the end of the line. The last line goes in at the end, which ends on the story's last sentence. In server dot c c, the text becomes lines, the story's lines, or the NPC's line when there's no story. The check for an empty text goes, and the world for the reply stays, as it's going to hold a few lines now. For each of the lines, the line is cleaned, as before, as the model's text could have control characters in it, and goes into the world under the NPC's name. The world goes to the player when it has any lines in it. As each line fits in two hundred bytes, clean chat lets it through, and a story arrives as a few lines from the storyteller.

34:35 The test for story lines goes in stories test dot c c, with string and vector, Abseil's join, to put the lines back together, and chat dot h, for the longest line. Breaks a story into lines of chat writes a story of ten sentences, in a loop, each saying the brave knight found a gold sword by the river, and adds then he, a sentence that the tokens ran out on. Four of those sentences fit in two hundred bytes, so the ten make three lines, each of them two hundred bytes at most, and each ending on a whole sentence. Joined back together, they're the ten sentences, without then he. And a story that ends in someone's quote keeps the closing quote, and drops what comes after it.

35:17 We run the server's tests for the stories, and both of them pass, and the client's tests with a filter for picking, which needs no window, as it's a ray and some balls, and it passes too.

35:38 Let's run it again, with a new account for alice. She walks up to the storyteller, clicks it, and asks for a story about a dragon, a castle and gold. It comes back after a moment, in her chat, in a few lines under the storyteller's name, each one ending on a whole sentence. The model learned from children's stories, so that's what it writes, and a model this small can leave one of the words out, or wander off from them after a few sentences.

Stories under load

36:06 With players asking for stories, we'd like to see them on our dashboard: how many, what became of them, and how long a player waits. In metrics dot h, add story counts a story by what became of it, told, refused, or failed, when the storyteller couldn't tell it, as its comment says. Observe story takes the time from when the player asked to when the story was told, which is what the player waits. Stories is a family of counters, by result, like the chat lines, and story seconds is a histogram. In metrics dot c c, the histogram gets buckets of its own, from a quarter of a second to ten seconds, where a story fails. The comment says why: a story takes seconds, where a tick takes milliseconds. The tick's buckets stop at a second, so a lot of the stories would land past the last one, where the histogram can't tell them apart. The counters are nanommo stories total, with stories asked for, told, refused or failed, as their help. The histogram is nanommo story duration seconds, the time from asking for a story to its telling, with the buckets for stories, which the add at the end registers with it. Add story adds one to the counter for its result, which add makes the first time it sees the result. Observe story puts the time in its bucket, in seconds, as a double, like the tick's, as Prometheus keeps its times in seconds, as floating point numbers. In server dot c c, each way that talk turns a player down counts as refused: an NPC that isn't there or doesn't talk, a player who's too far away, and words that make no prompt. The time we take the token at is when the player asked, and a player without a token is refused too. The lambda captures when they asked, along with the rest, and a story counts as told, with the time since then, while an error counts as failed. The counting comes before we look for the player's session, so that a story still counts when its player has left. The dashboard gets two panels by chat lines a minute: stories a minute, next to it, for each world and result, and waiting for a story, under it, with the median and the ninety-ninth percentile.

38:05 And our bots are going to ask for stories, so that the load has stories in it. In bot dot h, we include our generated protocol header, so that a bot can keep the storyteller in its land, once it has seen it. The storyteller is an optional protocol NPC, with its id and where it stands. And asking says whether its trip goes to the storyteller, for a story. In bot dot c c, iterator and vector come in, and Abseil's join. K words is an array of twelve words, a constexpr, under the lines that a bot says in chat, from sword, gate and skeleton to river, island and gold. Its comment says what they're for: what a bot asks the storyteller for a story about, three words from the game, which the model knows. K ask for a story is an eighth, so a bot asks for a story on one trip in eight. Random words picks three of the words with the standard library's ranges sample, which takes a random generator, and keeps them in their order, and joins them with spaces, like a player would type them. In receive, through a gate, a bot forgets the storyteller of the land it left, and stops asking, as the trip it was on was in that land. Then it looks through the NPCs in the world, and an NPC that talks is the storyteller of its land, which it keeps. NPCs come once a second, to the players near them, so the bots that start in the middle see the storyteller in their first second, and remember it as they wander off. In wander, the comment over it says that one trip in eight goes to the storyteller, once the bot has seen it, for a story. A Bernoulli distribution with k ask for a story decides it, when the bot has a storyteller, and the trip goes to where the storyteller stands, instead of a random place. When a trip ends, the bot stays around there for a while, as before. At the end of a trip to the storyteller, it stops asking, and asks the storyteller for a story about three random words, in the input it sends next, which also carries its next direction. A trip ends three metres from the storyteller, inside the four that the server allows, unless the bot got stuck on the way, and then the server turns it down, like it would a player.

40:15 Let's start the world with Prometheus, Grafana, the storyteller and a hundred bots, which ask for a story on one trip in eight. After a few minutes, stories a minute has the stories that Ember's bots ask for, which climb as the bots finish their trips. Most are told, and some are refused, from bots that got stuck at a lake on their way to the storyteller, and asked from where they stopped, which the server turns down as too far away. And waiting for a story has the median and the ninety-ninth percentile, from asking to telling, which stay well under the ten seconds after which a story fails, as the storyteller writes the stories that arrive together as a batch.

Docs

40:54 Last, the README, where the controls say to click the storyteller for a story, and the NPCs' section has the storytellers, with the bytes that the Meadows' four NPCs come to now. The list of metrics has the stories, and the bots' section says that a bot goes to the storyteller on one trip in eight, and in production, the storyteller stays on Compose's network. A new section on the storyteller says how to ask it for a story, and where the stories come from, mvGPT and its tag on the Hugging Face Hub, with the curl from earlier, to try mvGPT's server on its own. It goes through how a click picks the NPC, what the server checks, the prompt, and the words that the model knows, and how often a player can ask. It says that the server asks from threads of its own, so that the tick doesn't wait, and the storyteller can write sixteen stories at once, how a story becomes lines of chat, and the metrics. It ends on the setting for the storyteller's host, as any server with OpenAI's completions API could take its place, such as vLLM, as long as its model writes stories from words like ours.

Wrap-up

41:57 So, today one of our NPCs does something: a storyteller, which writes you a story from your words with mvGPT, whose server we copied in from my other series, and run in Compose next to the game. Your words go to the server in the input, where it checks how close you are, cleans the words, and limits how often you ask. Then it asks for the story from sixteen threads of its own, so that the tick doesn't wait, and the storyteller can write sixteen stories at once. On the client, a ray from the mouse picks the storyteller, and a window takes your words, and the story comes back as lines of chat, after a first try where it was too long for a single line, and went nowhere. The dashboard counts the stories and how long they take, and our bots ask for them too.

Playing it

43:32 Let me know in the comments what you'd like to see next. And if you want to follow along with the series, subscribe to the channel. I hope you find this useful, and thanks for watching.