I have spent most of my career on backend systems. Databases, services, the usual. I have read plenty of explanations of how large language models work and come away from each one able to repeat the explanation and not much else. So I built one.

Not a toy on a thousand lines of Shakespeare. The whole of English Wikipedia, 11 GB of it, on the RTX 3080 Ti that normally runs games, over twelve hours. The result is a 110 million parameter model that writes fluent, structurally correct, confidently wrong English.

It is a bad model. That turned out to be the useful part.

What the hardware was

One desktop: Ryzen 9 5950X, 128 GB RAM, and a single RTX 3080 Ti with 12 GB of VRAM. It dual boots into Windows for games, which is the only reason it exists.

Total electricity for the training run: about ten kilowatt hours, roughly a dollar twenty at Seattle rates.

The whole thing, in five minutes

Twelve hours of pipeline compressed 150x. One second of video is two and a half minutes of real time.

Five stages run in sequence. Four of them take 41 seconds of video between them. The fifth takes the remaining four minutes.

Stage one: an extract job

11 GB of English Wikipedia as parquet files, 41 of them, 6,407,814 articles. Three and a half minutes to download at 56 MB/s.

I used the HuggingFace wikimedia/wikipedia build rather than the raw XML dumps. The raw dumps are full of wikitext markup and parsing them is half a day of work that teaches nothing about language models.

That is the entire input. No labels, no question and answer pairs, no examples of a good response. Text.

Stage two: deciding what a document is

This stage threw away 481,673 articles, 7.5% of the corpus.

Two sentence stubs under 200 characters. Disambiguation pages. Articles whose titles start with "List of". Exact duplicates by content hash.

Every engineer knows garbage in, garbage out. This is sharper than that. Filtering is not cleaning here, it decides what the system will treat as normal. Leave the stubs in and the model learns that articles end after two sentences. Leave the list pages in and it learns to emit bullet points until it runs out of context.

5,926,141 articles survived, 92.5% of the input, 17.4 GB of prose.

Stage three: building a compression dictionary

This is the stage that will feel most familiar if you have built anything with a codec in it.

The model reads numbers, not letters. Something has to map text to numbers and back. The method is byte pair encoding: start with the 256 possible byte values, find the most frequent adjacent pair across the whole corpus, give that pair its own number, repeat.

I ran it 32,512 times, for a final vocabulary of 32,768.

The first merges are exactly what you would guess. t + h becomes th. Then th + e becomes the. The first merge collapsed 1,921,453 occurrences. By merge 2,775 a new entry was buying only 2,620 occurrences, and the curve from there is a straight line on a log axis.

That decay is the whole argument about vocabulary size. Every id costs a row in the embedding table and a row in the output layer forever, so past some point you are paying full price for almost nothing.

The result: common words become one number, uncommon ones stay in fragments. Seattle is two tokens. Kolhapur is five: K, ol, h, ap, ur. A model is fluent about one and clumsy with the other for this reason, and it has nothing to do with the model.

Final compression: 4.32 bytes per token. This stage took 70 minutes of wall clock and never touched the GPU. There is no learning in it. It is counting.

Stage four: encoding

4,389,677,049 tokens. 8.2 GB on disk as uint16.

uint16 holds 0 to 65535, which is why vocabulary size and storage are the same decision. A vocabulary above that doubles the corpus on disk.

Encoding is pure Python and runs at about 400k tokens a second, which would be three and a half hours for this corpus. Splitting the file into byte ranges snapped to line boundaries and running 30 workers brought it to 17 minutes. The output is byte identical to the serial version.

After this step the document structure is gone. No articles, no titles, no paragraphs. Wikipedia is one array, and training slides a window along it.

Stage five: a ten hour batch job

You have written this job. It has a loop, a metric, checkpoints every ten minutes, and a resume path for when the machine dies.

The configuration: 12 layers, 768 dimensions, 12 attention heads, 1024 token context, 110,926,080 parameters. 130,000 steps at 16,384 tokens per step, for 2,129,936,384 tokens total. That is about half the corpus, so the model never saw most of Wikipedia even once.

It held 57,900 tokens per second for the entire run with less than 0.3% variance, at 74°C and 358 W. The card sat at its power cap the whole time.

The metric is cross entropy loss. It started at 10.46. That is not arbitrary: there are 32,769 possible tokens and the natural log of 32,769 is 10.40, which is the score for guessing between them at random. It finished at 2.785, a perplexity of 16.

In plain terms, at the start it was choosing blindly between 32,769 options. At the end it was effectively choosing between 16.

What it writes

The same six prompts were sampled on a geometric schedule through training. Here is one of them, "Seattle is", at four points:

Step 135: that and, the government, or, and the which not to those and the two that, are of th

Step 1,842: the first generation of the United States, which is the only one of his first two po

Step 15,015: the area of the city, and makes up the city's easternmost and the city's most populous city

Step 130,000: an Australian Aboriginal language, spoken by a small number of people from around the world, that is, to the north of the island of Polynesia

The arc is clear. Noise, then word shapes, then a grammatical sentence that contradicts itself, then a grammatical sentence that is simply false.

Taking it for a spin

I put the model behind a small HTTP API with a playground on the front. It streams token by token at about 185 tokens a second.

Four prompts, no edits. The strip under the box shows how the tokenizer splits each one before the model sees it.

Ask it a question and it does not answer, because nothing has ever taught it that a question is a thing you answer:

What is the capital of France? / Description of the street. The main street runs through the streets of Paris from north to south.

Q: Who wrote Hamlet? A: / American LGBT novels / LGBT-related fantasy novels / Novels set in the 1560s

The second one fell into Wikipedia's category list format. It recognised "Hamlet" as an article topic and started producing the furniture that surrounds one.

Give it a statement to continue and it is fluent:

Kolhapur is / the main village located in Kalibapur taluka in the Kalibapur CD block in the Kalibapur district in the Indian state of West Bengal. Geography. Location. Kalibapur is located at .

That is structurally a real Indian village stub, down to the taluka and CD block vocabulary and the empty coordinate field those stubs have before someone fills them in. Kolhapur is in Maharashtra, not West Bengal, and Kalibapur does not exist.

A corpus defect that ended up in the model

"Kalibapur is located at ." with nothing after "at" is not the model failing. The HuggingFace parquet conversion strips Wikipedia's {{convert}} templates, so the training text contains sentences like "its coastline has a considerable length of ." with the number gone. In a sample of 2,567 documents I counted 69 occurrences.

The model learned faithfully that measurements in Wikipedia are blank.

What the model file actually is

The thing training produces is a file. Understanding what is inside it cleared up more for me than any diagram of a transformer.

ckpt.pt is 1,331 MB. It is a dictionary of named arrays, serialized. 101 entries, names like tok.weight and blocks.0.ln1.bias, each one a tensor of float32 numbers. There is no code in it, no graph, no logic. Loading it into the wrong model definition produces a shape mismatch and nothing else.

Where the 136,092,672 numbers in that file sit:

part	parameters	share
MLP blocks	56,623,104	41.6%
attention	28,311,552	20.8%
token embedding	25,166,592	18.5%
output projection	25,166,592	18.5%
position embedding	786,432	0.6%
layer norms	38,400	0.0%

Two things in that table surprised me. The MLPs hold twice what attention does, which is the opposite of where the attention-is-all-you-need reputation points. And the file contains 136 million parameters for a 110 million parameter model, because the output projection shares its weight matrix with the token embedding. The same 25 million numbers are stored twice under two names.

What you ship is not what you train

Of that 1,331 MB, the weights are 544 MB. The other 887 MB is Adam optimizer state: two full-size tensors for every parameter, carried so a crashed run can resume where it stopped. Inference reads none of it.

Dropping the optimizer state and the duplicated projection:

artifact	size
training checkpoint	1,331 MB	weights + optimizer + duplicate
inference, float32	444 MB	67% smaller
inference, bfloat16	222 MB	83% smaller

222 MB is the model. That is the whole thing, the complete output of twelve hours and two billion words of Wikipedia, small enough to attach to an email.

It needs one companion file: the 1.9 MB tokenizer. The checkpoint deals only in integer ids and has no idea what text is. Ship them apart and the weights are inert.

Bringing it up

Loading is unglamorous, which is the point. Timed on the training machine:

step	time
read the tokenizer	85 ms
construct the empty model	843 ms
read the file from disk	229 ms
copy weights into the model	58 ms
move to the GPU	225 ms
cold start	1,441 ms

The slowest step is building the empty model, before any trained value has been read. That is PyTorch allocating and randomly initialising 110 million numbers that are about to be overwritten.

Once up it holds 444 MB of VRAM and does nothing until asked.

How it answers

The serving path has four steps and no state.

1. Text becomes ids. Paris is the capital goes through the same BPE dictionary built in stage three and comes out as [20312, 318, 262, 3139].

2. One forward pass. Those ids go through the embedding table, then the 12 blocks, then the output projection. 2.5 ms. What comes back is shaped (1, 4, 32769): for every position in the input, a score for every one of the 32,769 tokens that could come next.

3. Pick one. Only the last position matters. Those 32,769 scores become probabilities, the lowest are discarded, and one token is drawn from what remains. That is what temperature and top-k control, and both live here, outside the model.

4. Append and repeat. The chosen token is added to the sequence and the whole thing runs again, now with five ids instead of four.

That last part is the bit I had wrong before building this. There is no state carried between tokens, and nothing streaming. Generating 100 tokens means 100 complete forward passes over a sequence that grows by one each time. 407 ms for 100 tokens, 246 tokens a second.

It is a pure function. Same input, same seed, same output, every time. Nothing is remembered between calls, and the model has no notion that a conversation is happening. What looks like memory in a chat interface is the entire conversation being re-sent and re-read from scratch on every message.

What I expected, and what training actually is

I went in assuming a model trained on Wikipedia would come out knowing roughly what is in Wikipedia.

When I read the Wikipedia article on the Fourier transform, I am not starting from nothing. I know what sound is because I have heard it. I know what a frequency is because I have turned a knob on a stereo and heard the bass come up. I did some calculus at university and have forgotten most of it, but enough shape remains that an integral sign does not stop me.

So the article lands on all of that at once, and somewhere in the middle there is a click: that is what the graphic equalizer on the stereo is doing. It splits the audio into its constituent frequencies so each band can be moved separately.

That click is the thing worth calling learning. One article, read once, connected to an object in my living room and to maths I half remember. The understanding transfers to things the article never mentioned, like why a cheap equalizer sounds blocky.

The model has none of that. Before training it is 110,926,080 random numbers. There is no sound, because it has never heard anything. There is no stereo. There is no sense that an integral is a kind of sum. Nothing for the article to land on.

What happens instead is mechanical. The article arrives as token ids. The model predicts the next one, gets it wrong, and all 110 million numbers are adjusted a very small amount in the direction that would have made it less wrong. Then the next token. Two billion times.

No part of that asks whether anything is true, or consistent with anything else, or connected to anything.

Here is my model on the same subject:

The Fourier transform is / a special case of the Fourier transform

A graphic equalizer works by / a computer programming language. It can be used to interact with an interdependent computer program.

Sound can be broken into / different shapes depending on the type of object.

Three questions about one idea. The first has the exact form of a mathematical definition and contains nothing. The second has the form of a software article. The third is a well made English sentence that means nothing in particular.

Someone who had the click answers all three the same way, and the third answer is the first answer. These three are unrelated to each other, because there is no idea underneath them that they are three views of.

A hypothesis I had, and why it was wrong

My first guess was that the model learns facts in proportion to how often it sees them, so common things would come out roughly right and rare things invented. That is testable.

Counting mentions across the 19.5 GB corpus:

entity	mentions	what the model says it is
United States	~2,355,000	a state in the Caribbean on the Gulf of Mexico
London	~1,109,000	a British writer, playwright and writer
Paris	~948,000	a French writer
Seattle	~141,000	a Canadian writer, playwright and writer
Mumbai	~47,000	a village in the tehsil of Chhattisgarh, India
Einstein	~32,000	a Canadian writer
photosynthesis	~4,200	a company under the name of J. C. Wilson and Sons
Kolhapur	~2,800	a village in the municipality of Jind, Haryana, India

Frequency predicts nothing. The United States appears over two million times and comes back as a Caribbean state. Kolhapur appears 2,800 times and comes back with identical confidence.

What the answers share is not the entity, it is the template. Almost all of them are a biography opening or a village stub. In a sample of 1,136,700 articles, 3.8% open as a village or town, 2.4% as a species, 1.7% as "is a [nationality] [occupation]". Five such templates cover 10.3% of all articles and they are far more uniform than the rest of the encyclopedia.

The model did not learn about the United States. It learned that a capitalised word followed by "is" is usually followed by a nationality and an occupation, or by a village in a district. That pattern appears hundreds of thousands of times. The United States article appears once.

What this cost

download	3.8 min
filtering	11.9 min
vocabulary	70 min
encoding	17 min
training	10.2 h
electricity	~10 kWh, about $1.20

The model is 1.3 GB as a checkpoint. It needs the 1.9 MB tokenizer file alongside it or the weights are inert, since the checkpoint deals only in integer ids.

What I would change

The context window was 1024 tokens against a mean article of 341 tokens, which is fine. An earlier run at 256 tokens could not fit the average article at all, and the model never saw one from start to end.

2.1 billion tokens against 110 million parameters is about 19 tokens per parameter, close to the Chinchilla ratio, so the balance was roughly right for the compute available. More of either would need more than one consumer card.

The {{convert}} artifact should be filtered in stage two.

What it would take to build the thing I originally wanted

The thing I actually wanted when I started: ask a question in plain English, get an answer that is correct according to Wikipedia, in proper English, with the source it came from.

What I have does none of that. Three separate gaps, and they are not the same size.

Gap one: it does not answer questions

This one is cheap. The model continues text because that is all it was ever shown. Supervised fine-tuning on a few thousand question and answer pairs teaches it the shape of a reply, and on this hardware that is twenty minutes of work.

It changes format, not knowledge. The answers would be confidently wrong in a more helpful tone.

Gap two: it does not know things

The obvious fix is a bigger model trained on more text. The arithmetic says otherwise, and it says it in a way I did not expect.

Our corpus is 4.39 billion tokens. At the usual compute-optimal ratio of about 20 tokens per parameter, that supports a model of roughly 220 million parameters. We trained 110.9 million on 2.13 billion tokens, 19 tokens per parameter, so we were already close to the best shape for the text available.

To go bigger:

target	tokens needed	text vs Wikipedia	time on this GPU
0.5B params	10B	2x	~2 weeks
1B params	20B	5x	~1 month
7B params	140B	32x	~4.8 years

Wikipedia runs out before the model gets large enough to hold it. A 7 billion parameter model needs thirty-two times more text than Wikipedia contains, so "train a bigger model on Wikipedia" is not a plan that terminates. You would be training on the open web, at which point you are no longer building what I described.

And models that size still get facts wrong. Scale moves the error rate. It does not give you correctness against a source.

Gap three: it cannot cite anything

This is the gap that decides the design. "Correct as per its source" requires knowing which source, and a model with facts baked into its weights cannot tell you. The weights are a blend of everything it read. There is no pointer back to the article a fact came from, because no article was stored.

No amount of training fixes that. It is structural.

What you would build instead

Stop asking the model to remember, and look the article up at query time.

1. Split the 5,926,141 articles into passages, roughly 24 million of them.
2. Embed each passage and build a search index. On this GPU that is a one-off cost of 4 to 16 hours depending on the embedding model, and the index is about 18 GB at 384 dimensions, which fits in RAM on this machine with room to spare.
3. On a question, retrieve the handful of passages that match.
4. Hand those passages to a model whose job is now reading comprehension rather than recall, and have it answer using only what it was given, quoting the article it used.

The model stops being the database and becomes the reader. Facts come from the retrieved text, so they are as correct as Wikipedia is, and the citation is simply whichever article you retrieved.

Where I would stop building

The retriever is buildable here. Chunking, embedding and indexing 24 million passages is a long batch job of exactly the kind this project already ran, and the index fits in memory.

The reader is not. Answering from a supplied passage without drifting into invention needs far more capability than 110 million parameters trained on 2 billion tokens provides, and getting there runs into the same arithmetic as gap two. That is the point where I would use an existing open model rather than train one, and the interesting work becomes retrieval quality: chunk size, what to embed, how many passages to pass, how to handle a question whose answer is spread across four articles.

Grounding the answer in a retrieved passage reduces invention. It does not eliminate it, since a model can still misread the text in front of it. The difference is that when it does, you have the passage it was given and can see that it did.

Code and the serving setup are in my homelab repo. It will run on any machine with 2 GB of VRAM, or on CPU at about a tenth of the speed.