Most engineers can call a language model API. Far fewer can explain what actually happens between hitting send and getting a response. Over the past few weeks I worked through the foundations of how large language models work, not the product features, but the machinery underneath. This is the field guide I wish I had at the start.
It moves in the order the pieces actually fit together: how text becomes something a model can process, what that model is, how it learned, and how it relates to the classical machine learning it is built on. By the end you should be able to trace a single message from end to end.
An LLM is a stateless next-token function
Strip away the chat interface and a language model is one thing: a function that takes a sequence of tokens and returns a probability distribution over what the next token should be. To generate text it samples one token, appends it to the input, and runs again. That loop, one token at a time, is the entire act of writing a response.
The most important consequence is that the model is stateless. It holds no memory between turns. When a chatbot seems to remember your earlier messages, the application is replaying the whole conversation back through the function on every single turn. There is no hidden notebook on the model side.
A second consequence is that the output is sampled, not looked up. A setting called temperature controls how adventurous that sampling is, which is why the same prompt can produce different answers. Coming from deterministic software, this is the first mental shift: a language model is a probabilistic function.
From text to numbers: tokens and tokenization
A model never sees your text. Before anything reaches it, the text is chopped into tokens, usually sub-word chunks, and each token is mapped to an integer ID. The model operates only on these integers.
The chopping is done by an algorithm, almost always Byte-Pair Encoding. It starts from raw bytes and repeatedly merges the most frequent adjacent pair into a new token. Common words collapse into a single token, rare words split into a few pieces, and nothing is ever truly unknown.
This explains a whole category of quirks. Take the word strawberry. It does not arrive as ten letters; it arrives as three integers.
"strawberry" -> 3 tokens
st -> 302
raw -> 1618
berry -> 19772
The model sees three integers, not ten letters.- Token
- The atomic unit a model reads and writes, usually a sub-word chunk represented as an integer ID.
- Tokenization
- The process that converts text into tokens, almost always using Byte-Pair Encoding (BPE).
Because the model sees integers rather than letters, asking it to count the r's in strawberry is genuinely hard, and the same root cause explains weak arithmetic and trouble reversing strings. Tokenization also has real cost implications: you are billed per token and your context budget is measured in tokens. Since the tokenizer learned mostly on English-heavy text, the same content in another language, or as JSON, fragments into far more tokens, which means more cost and less room for identical information.
The context window: a fixed working memory
The context window is the maximum number of tokens a model can process in one call. Everything shares this single budget: the system prompt, the full conversation history, any retrieved documents, your new message, and the reply it generates. It is the hard ceiling on the model's only memory.
A useful rule of thumb is about 750 words per 1,000 tokens. Modern windows range from around 128,000 tokens to a million or more. But a bigger window is not always better: models attend worse to information buried in the middle of a long context, an effect known as lost in the middle, so the usable context is smaller than the advertised number. This is the core reason retrieval, summarization, and memory systems exist, you feed the model the relevant slice rather than everything.
Embeddings: turning tokens into meaning
Integers alone are meaningless to compute on, so the first thing the model does is look each token ID up in an embedding table to get a vector. An embedding is a list of numbers that places a piece of text as a point in a high-dimensional space, arranged so that similar meanings sit close together.
This is what makes semantic search work. The query how do I reset my password can match a document about recovering account access even with no shared words, because both map to nearby points in the space. Similarity is measured by the angle between vectors, called cosine similarity. Text embeddings produced by dedicated models are the workhorse of retrieval and underpin most application work.
The Transformer: a stack of refinement layers
The architecture that processes those embedding vectors is the Transformer, the T in GPT. It came from the 2017 paper Attention Is All You Need and replaced the older sequential models that preceded it.
The shape is a stack of identical blocks. Each block does two things: self-attention, where every token gathers relevant information from the other tokens, and a feed-forward layer, where each token's representation is processed on its own. Stack that block dozens to over a hundred times, and a final layer turns the top position into a probability distribution over the next token.
The mental model is a stack of refinement passes. A token starts as its generic embedding and, layer by layer, becomes its meaning in this specific sentence. The reason the Transformer beat what came before is parallelism: it processes the whole input at once rather than one token at a time, which maps onto GPUs and is what let these models scale.
Attention: how each word finds what matters
Attention is the operation at the heart of the Transformer. It lets each token look across the sequence, score which other tokens are relevant, and pull in a weighted blend of their information. In the sentence the dog barked because it was scared, attention is how the word it gets linked to dog.
The mechanism is a soft lookup. Each token produces a Query for what it is looking for, a Key for what it offers, and a Value for the information it carries. A token compares its Query against every other token's Key to get relevance scores, turns those into weights, and takes a weighted sum of the Values. Several of these run in parallel, called multi-head attention. The cost is that every token attends to every other token, so the work scales with the square of the sequence length, which is why long context is expensive.
Parameters: the learned numbers that are the model
When a model is described as 8 billion parameters, that is the count of learned numbers inside it. Parameters are the weights, living in the embedding table, the attention projections, and the feed-forward layers. They are the model; the downloaded file is essentially a giant array of these values.
The most practical fact is that parameter count and precision tell you the memory you need to run a model.
memory (GB) ~= parameters (billions) x bytes per parameter
FP16 (2 bytes): 7B x 2 = ~14 GB
FP16 (2 bytes): 70B x 2 = ~140 GB
INT4 (0.5 bytes): 7B x 0.5 = ~3.5 GBQuantization, storing each number in fewer bits, is how large models are squeezed onto smaller hardware. Two distinctions matter: parameters are learned during training, while hyperparameters such as the learning rate are set by humans; and more parameters is not automatically better, since a well-trained smaller model often beats a poorly trained larger one.
Training versus inference: learn once, use forever
Training and inference are two very different jobs. Training is the expensive process of learning the parameters from data: the model predicts, the prediction is compared to the real answer to get a loss, gradients flow backward, and an optimizer nudges every weight. Repeated billions of times over weeks on large GPU clusters, this is where capability is baked in. Inference is using the frozen model to generate output, and it happens on every single request.
- Training: learns the weights, runs once or periodically, needs roughly four times the memory, optimized for throughput, hugely expensive.
- Inference: uses the frozen weights, runs on every request, needs about the model size plus a cache, optimized for latency.
The key takeaway for anyone building products: you live almost entirely on the inference side, and because the weights are frozen, the model only knows what was in its training data up to a cutoff. That single limitation is the entire reason retrieval-augmented generation exists, to inject fresh information into the context at answer time.
Open versus closed models
Models reach you in one of two ways. Closed models like GPT, Claude, and Gemini are accessed through an API with private weights. Open-weight models like Llama, Qwen, DeepSeek, and Mistral publish their weights so you can run and modify them yourself. A precise nuance: most open models are open-weight, not fully open source, because they release the weights but not the training data or code.
By 2026 the capability gap has narrowed to a small margin and open models are far cheaper per token, so the decision is no longer simply open or closed; it is three-way. Use a closed API for the fastest start and the frontier edge, an open model through a hosted provider for open economics without managing hardware, and self-hosted open weights when data cannot leave your environment or when sustained volume justifies the infrastructure.
The classical ML underneath
None of this is separate from classical machine learning; it is classical machine learning at scale. The next-token objective is a giant classification problem where the text supplies its own labels, which is why pretraining is called self-supervised. The old fundamentals still apply: models can overfit, which is why you hold out a test set, and evaluation needs more than accuracy, since on imbalanced data a model that always predicts the majority class can be 99 percent accurate and useless. Two metrics carry most of the weight, and they return later when evaluating retrieval and AI systems.
- Precision
- Of everything the model flagged as positive, the fraction that was actually positive. Measures whether you can trust an alarm.
- Recall
- Of everything that was actually positive, the fraction the model caught. Measures whether anything was missed.
Putting it together: what happens when you send a message
With the pieces in place, the lifecycle of a single message is clear. You hit send, and the application assembles the entire context, a system prompt, the whole conversation history, and your new message, wrapped in special role markers. For an empty conversation that scaffolding alone looks like this:
<|im_start|>system<|im_sep|>You are a helpful assistant<|im_end|>
<|im_start|>user<|im_sep|>How are you?<|im_end|>
<|im_start|>assistant<|im_sep|>The final assistant turn is left open on purpose: it is the cue for the model to start generating. That whole context is tokenized into integers, each integer becomes an embedding vector, and the sequence flows through the Transformer stack in parallel during the prefill phase. The model then generates the reply one token at a time in the decode phase, which is what you see streaming onto the screen, until it emits a stop token. Throughout, it uses only its frozen parameters. Every reply is the same loop: re-read the whole conversation, predict the next token, repeat.
Why the foundations matter
These concepts are not trivia. They are the reasons behind nearly every practical decision that comes later: why context management and retrieval matter, why cost scales the way it does, why some tasks suit a small model, and how to judge whether any of it actually works.
This is the foundation. The next stage is building on top of it: prompting, retrieval-augmented generation, agents, evaluation, and the production engineering that turns a demo into a system. This article is one checkpoint in a longer effort to learn applied AI engineering in public. More to come.