How Language Generation Works: From Tokens to Transformers

An autoregressive language model generates text by repeatedly predicting a distribution over the next token. Understanding that loop requires separating tokenization, learned representations, and the computation that turns a context into a prediction.

Tokenize text into a sequence of IDs from a fixed vocabulary.
Embed those IDs into vectors and incorporate position information.
Predict using transformer blocks, then select a token and repeat.

The figures are schematic illustrations, not measurements from a trained model. Their TikZ source defines the example token IDs, geometry, and attention arrows.

1. Representing Text as Tokens

A text model needs a numerical representation of its input. Transformers can process variable-length sequences of vectors with a fixed feature dimension; tokenization determines how text is divided into sequence elements.

A naïve approach would be to represent each word as a one-hot vector. This immediately runs into two problems:

Word-level vocabularies are extremely large, leading to impractically high-dimensional vectors
Rare or unseen words cannot be represented meaningfully

Character- or byte-level representations use smaller vocabularies but longer sequences. The model must learn larger linguistic units across more positions, increasing sequence-processing costs.

Subword tokenization offers a common compromise, although character- and byte-level models are also possible.

Comparing character-level, word-level, and subword tokenization — Figure 1 — Illustrative segmentations of “unbelievable.” Character models must learn larger units across more positions; “weak higher-level structure” here refers to the input units, not a limit on what the model can learn. Actual subword splits depend on the tokenizer.

2. Tokenization as Statistical Compression

Tokenization maps raw text into a sequence of discrete symbols drawn from a fixed vocabulary. Tokens may represent bytes, characters, word fragments, whole words, or punctuation; they are not uniformly sized linguistic units.

Common tokenizer algorithms choose units using corpus statistics. Their design balances:

Compactness (frequent substrings get short representations)
Coverage (rare words can be decomposed)
Decoding behavior (normalization or unknown-token handling can prevent exact recovery of the original text)

For a standard BPE pipeline, tokenizer training precedes language-model training and the vocabulary is then fixed. Sennrich, Haddow and Birch describe a foundational subword approach.

Once tokenized, text becomes a sequence of integer token IDs. These IDs are not meaningful by themselves; they are merely indices into a learned embedding table.

3. Embeddings: From Discrete Tokens to Geometry

Embeddings as a Lookup Table

An embedding layer is a learned matrix , where is the vocabulary size and is the embedding dimension.

Each token ID indexes a row of this matrix, producing a dense vector. At this stage, there is nothing inherently semantic about the vectors—they are simply parameters to be optimized.

Meaning arises only through training.

Token IDs are used to look up embedding vectors from a learned matrix — Figure 2 — Example token IDs select rows of the learned embedding table, preserving sequence order. Practical implementations gather rows directly instead of constructing one-hot vectors.

---

Geometry Emerges from the Training Objective

A good embedding model is one in which linguistic regularities are reflected as geometric regularities in the embedding space. This structure is not enforced directly. Instead, it emerges because the training objective rewards representations that make prediction easier.

To build intuition, consider an idealized two-dimensional embedding space:

One axis loosely corresponds to gender
Another corresponds to some relational or semantic role

In such a space, vectors for "man" and "woman" would differ primarily along the gender axis, while "husband" and "wife" would show a similar displacement. The important property here is not absolute position, but relative geometry—differences and directions encode relationships.

Real embedding spaces need not have interpretable coordinate axes. Useful relationships can be distributed across many dimensions, and analogy-like directions are empirical patterns rather than architectural guarantees.

This is a desired emergent property, not a guarantee. Embedding quality is ultimately empirical and is evaluated by how well geometric proximity supports generalization in downstream tasks.

Toy example: embedding geometry with illustrative axes — Figure 3 — A constructed two-dimensional analogy: similar displacements represent a relationship between word pairs. The axes and points are illustrative, not fitted embeddings.

4. Encoder, Decoder, and Encoder–Decoder Architectures

Once tokens are embedded, different model architectures determine how sequences are processed and what the model is optimized to do. Most modern language models fall into one of three categories.

Comparison of encoder-only, encoder-decoder, and decoder-only architectures — Figure 4 — Common transformer arrangements: an encoder produces contextual representations, an encoder–decoder conditions generation on encoded input, and a decoder predicts the next token from a prefix.

Encoder-only models (e.g. Jina v3)

Encoder-only architectures process the entire input sequence simultaneously using bidirectional context. Every token can attend to every other token, allowing the model to build rich, global representations.

Embedding models such as Jina v3 fall into this category. Their training targets useful text representations for tasks such as retrieval and similarity comparison.

Because encoder-only models see full context in both directions, they are well suited for:

Semantic search and retrieval
Clustering and similarity comparison
Reranking and matching tasks

In this use, the encoder supplies representations rather than an autoregressive text-generation loop.

Encoder–decoder models (e.g. T5)

Encoder–decoder architectures are designed for sequence-to-sequence problems. An encoder first processes the input sequence into an internal representation. A decoder then generates an output sequence conditioned on that representation.

A canonical example is T5, which frames its tasks—translation, summarization, question answering—as text-to-text transformations.

This architecture is especially effective when:

Input and output lengths differ
The task requires rephrasing rather than continuation
There is a clear source → target structure

Conceptually, encoder–decoder models transform one sequence into another.

Decoder-only models (e.g. GPT-3)

Decoder-only architectures generate text autoregressively: each token is predicted conditioned only on previous tokens. There is no separate encoder; the same stack is used for both conditioning and generation.

GPT-3 is an example of this design, trained on large text corpora with a next-token prediction objective and evaluated on tasks specified through prompts.

Their strengths include:

Open-ended text generation
In-context learning via prompting
Flexible adaptation without task-specific heads

A useful way to think about the three architectures is:

Encoders represent
Encoder–decoders translate
Decoders generate

These labels describe common uses, not exclusive capabilities: decoder-only models can also translate, and encoder–decoder models can generate open-ended text.

5. Transformers and the Attention Mechanism

A transformer block combines attention with a feed-forward network, residual connections, and normalization. Attention mixes information across positions; the feed-forward network transforms each position's representation.

For hidden-state matrix , learned projections form queries , keys , and values . One attention head computes:

The softmax acts across keys for each query. In a causal decoder, the mask excludes future positions. Scores compare learned queries and keys, so they need not coincide with semantic similarity between the original token embeddings.

Position information, supplied through positional encodings or position-dependent transformations, lets the model represent token order. Multiple heads use different learned projections; particular linguistic specializations are possible, not guaranteed.

During training, the known sequence and causal mask allow predictions at many positions to be computed in parallel. During autoregressive generation, each new token depends on the tokens already selected. Vaswani et al., Attention Is All You Need describes the original architecture.

Attention as dynamic relevance routing: stronger connections indicate higher attention weights — Figure 5 — Schematic attention links for “apologized.” Arrow widths are chosen to illustrate unequal weights; they are not measured attention scores or evidence of a particular head’s linguistic role.

6. Putting It All Together

The final hidden state is projected to vocabulary logits . Softmax converts these unnormalized scores into probabilities:

Training minimizes the negative log-probability assigned to the observed next tokens. At inference, a decoding rule selects a token—for example, sampling from the distribution or choosing its largest-probability entry. The token is appended to the context, and the process repeats until a stop token or length limit is reached.

Temperature rescales logits before softmax, while top-k or top-p sampling restricts the candidate set. These choices change generated text without changing the trained model weights.

End-to-end flow from raw text to next token prediction — Figure 6 — One prediction step. The final arrow abbreviates softmax and token selection. Generation feeds the selected token back into the context and repeats the pipeline.

With a fixed tokenizer, the embeddings, attention projections, feed-forward networks, and output parameters are trained together. Learned structure comes from that prediction objective; neither interpretable embedding axes nor human-like understanding follows from the architecture alone.