COBURNICUSAt the end of the first episode, AdamAI was a tiny model chatting in its own terminal. It had about 9.5 million parameters, the adjustable numbers where a model stores what it learns, and only a megabyte of text to read. Mostly it memorized. I promised we would grow it for real next.
AdamAI Genesis is the same design scaled up to the size of GPT-2 small: about 114 million parameters and a reading budget of about ten billion tokens. These notes cover the check-in at step 13,500, on November 5, when it had read about 3.5 billion tokens. That’s a little over a third of the run, and it’s still training.
We watch this closely because people call the inside of a trained AI a black box, and AdamAI exists to open one. We save copies of its brain along the way (seventeen so far) and quiz it every 250 steps. Once we know when an ability shows up, we can go looking for where it lives in the network.
The one task
Training repeats a single task billions of times: read some text, hide the next token, guess it. Tokens are whole words or chunks of words, and Genesis knows about 50,000 of them.
The score is the loss, measured in nats: how surprised the model was by the right answer. Backpropagation traces each miss back through the network and works out which way to nudge every weight. That set of nudges is the gradient, and the optimizer, AdamW, takes the step.
The step size warms up over the first 500 steps, because big steps on a random brain just thrash around, then eases off along a cosine curve. Each step learns from about a quarter of a million tokens, and the full run is about 38,000 steps.
The food is FineWeb-Edu, a large collection of educational web pages from Hugging Face. That works out to about 88 tokens per parameter, more than four times the usual rule of thumb of twenty from DeepMind’s Chinchilla paper. We did that on purpose. Small models keep improving the more they read.
Birth
A brain that knows nothing should spread its odds evenly, like a die with 50,000 sides. That scores a loss of about 10.8. When we pressed go on the night of November 2, Genesis scored about 11.
Asked the capital of France, its top guess was “cess.” Paris sat around 26,000th place. Ten steps later, its favorite answer to almost everything was “the.”
At step 50 the meaning of life came out as dashes and line breaks. That looked worse, but the loss kept falling. The data is full of bulleted lists, and in its first hour, Genesis learned what text looks like before it learned what words mean.
Grammar before sense
By step 200 the samples were sentences. “Photosynthesis is the process of this page.” Grammatical, and close to meaningless.
We test grammar with a probe, a small quiz. Given “The keys to the cabinet,” does it prefer “are” or “is”? The trap is the single cabinet sitting right before the verb. Early on it fell for it. By step 500, about two hours in, it had switched to “are,” and all six grammar tests have passed ever since.
The copying circuit
Genesis has 144 attention heads, small readers that each decide which earlier words to look back at. At step 500, one head in layer 7 started spending most of its attention on the word just before. That’s a previous-token head.
By step 750, a head one layer up learned to find the last time the current word appeared and look at what came right after it. That’s an induction head, and it builds on the first head’s notes. Our copying test jumped right at that checkpoint. Catherine Olsson and colleagues at Anthropic described this same two-head circuit in 2022, forming in the same order. In plain terms, the model learned that when something happened before, it will probably happen again.
Facts arrive, and wobble
We ask ten simple questions and check where the right answer ranks among all 50,000 tokens. Number one means it knows it.
The first fact was “water is made of hydrogen and oxygen,” at step 750. By step 4,750, half of the ten were at the top. Some misses had the category right: after “Monday, Tuesday,” it guessed “March.”
It also forgets. It learned “one, two, three, four,” then lost it a few hundred steps later. The leading guesses are nearly tied, and every step rewrites the network a little. These days seven or eight of the ten usually sit at number one.
One thing that didn’t make the film: our lab notebook predicted that gold’s symbol, Au, would be the last fact to arrive. It came in at step 10,000, ahead of Jupiter at 11,250, and then slipped back to 12th place by step 13,500.
Knowing it, then talking itself out of it
The logit lens is a trick for reading the model’s guess partway up the stack, as if we stopped it at a given layer. For most layers there’s almost nothing. Then the answer lights up near the top. Tokyo is the clear favorite after layer ten, at an estimated 93 percent, and the last layer talks it down to about one in four. Our guess is that the final layer hedges toward other ways to finish the sentence. We haven’t tested that yet.
What went wrong
Twelve minutes into the first launch, a networking script restarted the machine’s Linux environment and killed training before its first save. We started over. The new run’s first score matched the old one to four decimal places, so the run is reproducible.
A sneakier bug cost about thirty percent of our speed: after a restart, the script loaded its save file into working memory and never released it, which pushed the finish line from about six days to more than eight. Neither problem hurt the model. They only cost time.
Where it stands
The loss is easier to feel as perplexity, roughly how many words the model is torn between on each guess. Early on it was hundreds. At this check-in it’s about two dozen. Scores on training text and unseen test text stay together, so it isn’t memorizing.
We also scored OpenAI’s real GPT-2 small, from 2019, on the same test pages, and Genesis passed it about 25 hours in. That’s home turf, since these pages are the kind Genesis studies. The neutral test comes at the end: HellaSwag, which asks a model to pick the sensible ending for a sentence. GPT-2 small scored about 29 percent there, and random guessing gets 25.
After Genesis comes a ladder of bigger models, changing one thing per rung, up to Exodus at about three times the size.
The open question for episode three: we know when grammar, copying and facts showed up. Will the saved brain snapshots show us where they live, and will the last layer’s habit of talking itself out of right answers turn out to be hedging or something we haven’t thought of?
Sources
- Penedo et al., The FineWeb Datasets (2024): https://arxiv.org/abs/2406.17557
- Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2, 2019)
- Loshchilov and Hutter, Decoupled Weight Decay Regularization (AdamW, 2017): https://arxiv.org/abs/1711.05101
- Kingma and Ba, Adam (2014): https://arxiv.org/abs/1412.6980
- Olsson et al., In-context Learning and Induction Heads (2022): https://arxiv.org/abs/2209.11895
- nostalgebraist, interpreting GPT: the logit lens (2020): https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens
- Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla, 2022): https://arxiv.org/abs/2203.15556
- Zellers et al., HellaSwag (2019): https://arxiv.org/abs/1905.07830
- All run numbers come from AdamAI Genesis’s own training logs and probes. Timings and costs for future runs are estimates.
If you’d like to help fund the bigger runs, that’s at https://coburnicusmind.com/compute.