Growing a Brain · Episode 1

Lab Notes: Building a Language Model From an Empty File

From 65 characters and a lookup table that writes gibberish to a small GPT that writes in the shape of a play.

November 3, 2026 · Adam Coburn
Subscribe on YouTube

People call the inside of an AI a black box. You type something in, an answer comes out, and nobody can really tell you what happened in between.

Growing a Brain is my attempt to open that box. The cleanest way I could think of was to build one myself, starting from an empty file and writing every piece by hand. If we build it, we can stop at any point and look inside. The model is called AdamAI, and this first episode is the build: raw text in, a model that writes on its own out.

These are the lab notes behind it. The numbers below are what our runs printed, rounded, so treat them as estimates from one small model.

Step one: text becomes numbers

A neural network can’t read letters. It only does math on numbers. So the first piece is a tokenizer, the part that turns text into numbers and back again.

We started with the simplest kind, character-level. Take every unique character in the training text, sort them, give each one a number. Our whole vocabulary came out to 65 characters. “Hello there!” turns into a list of numbers, and decoding the list gives back exactly the same text. Everything else in the series sits on top of that round trip.

The training text was a small collection of Shakespeare’s plays, about one megabyte.

The first model hits a floor

The first real model is a bigram model. It guesses the next character from only the current one, using a single lookup table. The model itself barely matters. What matters is the training loop around it: a forward pass to get scores, a loss to measure how wrong the guess was, backpropagation to work out which way to adjust, and an optimizer to nudge the numbers. Every model after this uses the same loop.

The loss (the model’s measure of how surprised it is by the right answer) started around 4.7, about what random guessing across 65 characters looks like. It fell to about 2.5 and went flat. That floor was the first lesson. With one character of context, there is only so much you can know about what comes next.

The output was gibberish. Still, it had picked up that a script puts the speaker’s name first and their lines underneath, and its made-up words were about the right length.

Letting tokens look back

To get past the floor, each character needs to see the ones before it. That is self-attention, the idea transformers are built on.

Every token puts out three things. A query: what am I looking for. A key: what do I have. A value: what I’ll pass along. Matching a query against the keys gives a score for how relevant each earlier token is. We divide by the square root of the head size so the scores don’t grow too large and swamp the softmax, the step that turns scores into weights adding up to one. You can watch the variance drop from about 16 back to about 1 when you do it. Then we hide the future so no token can peek ahead.

The result is a triangle of weights, and the weights are uneven. The model decides for itself which parts of its past to pay attention to.

AdamAI becomes a GPT

Stack those attention heads into transformer blocks, add a small feed-forward network so each token has room to think, wrap each piece in normalization and a shortcut connection, and you have a GPT. Ours had six blocks and about 10.8 million parameters (the adjustable numbers the model learns). Same design as the big models, at a tiny size.

The loss fell from about 4.3 to below 1.5, well under the bigram’s floor of about 2.46. And the writing changed. Real words, speaker tags, line breaks, the rhythm of blank verse. It produced names like Menenius and Hermione, real characters from the plays. Nobody taught it English, drama or spelling. It learned them from one megabyte of text by guessing the next character over and over.

The overfitting lesson

Next we built byte-pair encoding, the tokenizer real GPTs use. It starts from 256 raw bytes and keeps merging the pair of tokens that appears together most often. Its first merges were space-t, h-e and space-a. By merge 100 it had found the whole word “have.” The text shrank from about 1.1 million characters to roughly half as many tokens.

Then we trained the same GPT on those tokens, and the training loss kept falling while the validation loss (the score on text it hasn’t seen) bottomed out around step 1,500 and climbed. That split is overfitting: the model had moved from learning the language to memorizing its training text. Measured fairly, in bits per character, the new tokenizer landed around 2.2, about tied with the old one. At this size it didn’t win.

We then added the upgrades newer open models like GLM and Llama use: rotary position embeddings (RoPE, which encodes how far apart two tokens are by rotating them), RMSNorm (a cheaper normalization), SwiGLU (a gated feed-forward layer) and grouped-query attention (several query heads sharing one set of keys and values). The model got smaller, about 9.5 million parameters, and hit the old model’s best validation loss of about 1.48 in roughly a thousand steps instead of about 4,500. Then it memorized our one megabyte even faster.

What didn’t make the film

The overfitting has a simple rule of thumb behind it. The Chinchilla scaling work suggests about 20 tokens of training data per parameter. For our roughly 9.47 million parameters, that works out to about 190 million tokens. We gave it about one million. In our notes we compared it to Qwen2.5-1.5B: about 160 times the parameters and around 18 trillion tokens of data, something like ten million times what AdamAI read. The design uses the same parts as Llama and Qwen. The size is about 1/100,000 of a real LLM.

That gap is why Episode 2 moves to real data and a real size.

Where it ended

The episode ends with AdamAI in its own terminal app, streaming replies one character at a time, with live dials for temperature (how adventurous its word choices are) and top-k (how many candidates it is allowed to pick from). Prompt it with “ROMEO” and it writes back blank verse.

The open question

We can now watch every number this model produces. What I still don’t know is whether those numbers will tell us anything we can read once the model is big enough to be interesting. Episode 2 starts finding out.

Sources

AdamAIWhat it is and where it's going Fund the computeHelp pay for bigger training runs All episodesThe whole series, newest first